[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260808 20:12:21.724236 10336 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.10.24.62:33591
I20260808 20:12:21.726503 10336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 20:12:21.727661 10336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 20:12:21.756021 10347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 20:12:21.756196 10343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 20:12:21.756923 10351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 20:12:21.758706 10336 server_base.cc:1034] running on GCE node
I20260808 20:12:23.024783 10336 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 20:12:23.025046 10336 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 20:12:23.025207 10336 hybrid_clock.cc:648] HybridClock initialized: now 1786219943025190 us; error 0 us; skew 500 ppm
I20260808 20:12:23.026261 10336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 20:12:23.036100 10336 webserver.cc:460] Webserver started at http://127.10.24.62:43961/ using document root <none> and password file <none>
I20260808 20:12:23.037315 10336 fs_manager.cc:359] Metadata directory not provided
I20260808 20:12:23.037508 10336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 20:12:23.038003 10336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 20:12:23.045557 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786219941176655-10336-0/minicluster-data/master-0-root/instance:
uuid: "7960e0673c6f456db25b0283a3a3732c"
format_stamp: "Formatted at 2026-08-08 20:12:23 on dist-test-slave-ncp9"
I20260808 20:12:23.053967 10336 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260808 20:12:23.060019 10359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 20:12:23.061403 10336 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260808 20:12:23.062582 10336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786219941176655-10336-0/minicluster-data/master-0-root
uuid: "7960e0673c6f456db25b0283a3a3732c"
format_stamp: "Formatted at 2026-08-08 20:12:23 on dist-test-slave-ncp9"
I20260808 20:12:23.062911 10336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786219941176655-10336-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786219941176655-10336-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786219941176655-10336-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 20:12:23.118427 10336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 20:12:23.120097 10336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 20:12:23.120550 10336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 20:12:23.197443 10336 rpc_server.cc:307] RPC server started. Bound to: 127.10.24.62:33591
I20260808 20:12:23.197510 10435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.24.62:33591 every 8 connection(s)
I20260808 20:12:23.208329 10436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:23.233948 10436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c: Bootstrap starting.
I20260808 20:12:23.241189 10436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:23.242913 10436 log.cc:824] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c: Log is configured to *not* fsync() on all Append() calls
I20260808 20:12:23.247291 10436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c: No bootstrap required, opened a new log
I20260808 20:12:23.270716 10436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7960e0673c6f456db25b0283a3a3732c" member_type: VOTER }
I20260808 20:12:23.271214 10436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:23.271440 10436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7960e0673c6f456db25b0283a3a3732c, State: Initialized, Role: FOLLOWER
I20260808 20:12:23.272435 10436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7960e0673c6f456db25b0283a3a3732c" member_type: VOTER }
I20260808 20:12:23.273000 10436 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 20:12:23.273241 10436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 20:12:23.273548 10436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:23.277431 10436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7960e0673c6f456db25b0283a3a3732c" member_type: VOTER }
I20260808 20:12:23.278244 10436 leader_election.cc:302] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7960e0673c6f456db25b0283a3a3732c; no voters:
I20260808 20:12:23.280184 10436 leader_election.cc:288] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 20:12:23.280515 10441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:12:23.282883 10441 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c [term 1 LEADER]: Becoming Leader. State: Replica: 7960e0673c6f456db25b0283a3a3732c, State: Running, Role: LEADER
I20260808 20:12:23.283834 10441 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7960e0673c6f456db25b0283a3a3732c" member_type: VOTER }
I20260808 20:12:23.284946 10436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 20:12:23.299391 10447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7960e0673c6f456db25b0283a3a3732c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7960e0673c6f456db25b0283a3a3732c" member_type: VOTER } }
I20260808 20:12:23.300112 10447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c [sys.catalog]: This master's current role is: LEADER
I20260808 20:12:23.299554 10444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7960e0673c6f456db25b0283a3a3732c. Latest consensus state: current_term: 1 leader_uuid: "7960e0673c6f456db25b0283a3a3732c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7960e0673c6f456db25b0283a3a3732c" member_type: VOTER } }
I20260808 20:12:23.300809 10444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c [sys.catalog]: This master's current role is: LEADER
I20260808 20:12:23.304096 10458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 20:12:23.314198 10458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 20:12:23.316100 10336 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260808 20:12:23.331826 10458 catalog_manager.cc:1347] Generated new cluster ID: f989da35d2b84faa936c78e869e31ec4
I20260808 20:12:23.332044 10458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 20:12:23.350116 10458 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 20:12:23.352847 10458 catalog_manager.cc:1504] Loading token signing keys...
I20260808 20:12:23.369977 10458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c: Generated new TSK 0
I20260808 20:12:23.371209 10458 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 20:12:23.382236 10336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 20:12:23.391209 10467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 20:12:23.393090 10469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 20:12:23.396288 10471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 20:12:23.396775 10336 server_base.cc:1034] running on GCE node
I20260808 20:12:23.397738 10336 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 20:12:23.397997 10336 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 20:12:23.398206 10336 hybrid_clock.cc:648] HybridClock initialized: now 1786219943398187 us; error 0 us; skew 500 ppm
I20260808 20:12:23.398963 10336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 20:12:23.403147 10336 webserver.cc:460] Webserver started at http://127.10.24.1:46685/ using document root <none> and password file <none>
I20260808 20:12:23.403815 10336 fs_manager.cc:359] Metadata directory not provided
I20260808 20:12:23.404057 10336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 20:12:23.404392 10336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 20:12:23.406145 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786219941176655-10336-0/minicluster-data/ts-0-root/instance:
uuid: "7858bb58840c4165b68c2f114ae20263"
format_stamp: "Formatted at 2026-08-08 20:12:23 on dist-test-slave-ncp9"
I20260808 20:12:23.412978 10336 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 20:12:23.417857 10477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 20:12:23.418627 10336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260808 20:12:23.418951 10336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786219941176655-10336-0/minicluster-data/ts-0-root
uuid: "7858bb58840c4165b68c2f114ae20263"
format_stamp: "Formatted at 2026-08-08 20:12:23 on dist-test-slave-ncp9"
I20260808 20:12:23.419340 10336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786219941176655-10336-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786219941176655-10336-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786219941176655-10336-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 20:12:23.438452 10336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 20:12:23.440165 10336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 20:12:23.443269 10336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 20:12:23.448662 10336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 20:12:23.448980 10336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260808 20:12:23.449290 10336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 20:12:23.449501 10336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 20:12:23.551766 10336 rpc_server.cc:307] RPC server started. Bound to: 127.10.24.1:39783
I20260808 20:12:23.551810 10564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.24.1:39783 every 8 connection(s)
I20260808 20:12:23.560328 10336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 20:12:23.571894 10573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 20:12:23.573271 10574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 20:12:23.584082 10336 server_base.cc:1034] running on GCE node
W20260808 20:12:23.586315 10576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 20:12:23.587466 10336 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 20:12:23.587742 10336 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 20:12:23.587960 10336 hybrid_clock.cc:648] HybridClock initialized: now 1786219943587942 us; error 0 us; skew 500 ppm
I20260808 20:12:23.588836 10336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 20:12:23.589594 10565 heartbeater.cc:347] Connected to a master server at 127.10.24.62:33591
I20260808 20:12:23.590013 10565 heartbeater.cc:464] Registering TS with master...
I20260808 20:12:23.590881 10565 heartbeater.cc:511] Master 127.10.24.62:33591 requested a full tablet report, sending...
I20260808 20:12:23.594473 10389 ts_manager.cc:194] Registered new tserver with Master: 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783)
I20260808 20:12:23.595378 10336 webserver.cc:460] Webserver started at http://127.10.24.2:34531/ using document root <none> and password file <none>
I20260808 20:12:23.596091 10336 fs_manager.cc:359] Metadata directory not provided
I20260808 20:12:23.596349 10336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 20:12:23.596698 10336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 20:12:23.597890 10389 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40910
I20260808 20:12:23.598205 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786219941176655-10336-0/minicluster-data/ts-1-root/instance:
uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
format_stamp: "Formatted at 2026-08-08 20:12:23 on dist-test-slave-ncp9"
I20260808 20:12:23.604846 10336 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 20:12:23.609120 10584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 20:12:23.609877 10336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 20:12:23.610116 10336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786219941176655-10336-0/minicluster-data/ts-1-root
uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
format_stamp: "Formatted at 2026-08-08 20:12:23 on dist-test-slave-ncp9"
I20260808 20:12:23.610353 10336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786219941176655-10336-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786219941176655-10336-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786219941176655-10336-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 20:12:23.644351 10336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 20:12:23.645587 10336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 20:12:23.648098 10336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 20:12:23.651480 10336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 20:12:23.651661 10336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 20:12:23.651890 10336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 20:12:23.652029 10336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 20:12:23.753892 10336 rpc_server.cc:307] RPC server started. Bound to: 127.10.24.2:43407
I20260808 20:12:23.754011 10683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.24.2:43407 every 8 connection(s)
I20260808 20:12:23.761044 10336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 20:12:23.769680 10691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 20:12:23.777535 10692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 20:12:23.781414 10686 heartbeater.cc:347] Connected to a master server at 127.10.24.62:33591
I20260808 20:12:23.781821 10686 heartbeater.cc:464] Registering TS with master...
I20260808 20:12:23.782618 10336 server_base.cc:1034] running on GCE node
I20260808 20:12:23.782835 10686 heartbeater.cc:511] Master 127.10.24.62:33591 requested a full tablet report, sending...
W20260808 20:12:23.783362 10694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 20:12:23.784564 10336 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 20:12:23.784865 10336 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 20:12:23.785089 10336 hybrid_clock.cc:648] HybridClock initialized: now 1786219943785072 us; error 0 us; skew 500 ppm
I20260808 20:12:23.785934 10336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 20:12:23.785898 10389 ts_manager.cc:194] Registered new tserver with Master: ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407)
I20260808 20:12:23.788100 10389 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40916
I20260808 20:12:23.789206 10336 webserver.cc:460] Webserver started at http://127.10.24.3:40775/ using document root <none> and password file <none>
I20260808 20:12:23.789824 10336 fs_manager.cc:359] Metadata directory not provided
I20260808 20:12:23.790050 10336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 20:12:23.790308 10336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 20:12:23.791474 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786219941176655-10336-0/minicluster-data/ts-2-root/instance:
uuid: "186c827b8b48447b94beb413077d76fe"
format_stamp: "Formatted at 2026-08-08 20:12:23 on dist-test-slave-ncp9"
I20260808 20:12:23.796322 10336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260808 20:12:23.802927 10700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 20:12:23.804036 10336 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260808 20:12:23.804371 10336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786219941176655-10336-0/minicluster-data/ts-2-root
uuid: "186c827b8b48447b94beb413077d76fe"
format_stamp: "Formatted at 2026-08-08 20:12:23 on dist-test-slave-ncp9"
I20260808 20:12:23.804733 10336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786219941176655-10336-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786219941176655-10336-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786219941176655-10336-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 20:12:23.827168 10336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 20:12:23.828332 10336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 20:12:23.829658 10336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 20:12:23.832718 10336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 20:12:23.832942 10336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 20:12:23.833166 10336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 20:12:23.833315 10336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 20:12:23.932749 10336 rpc_server.cc:307] RPC server started. Bound to: 127.10.24.3:37651
I20260808 20:12:23.932888 10798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.24.3:37651 every 8 connection(s)
I20260808 20:12:23.951923 10799 heartbeater.cc:347] Connected to a master server at 127.10.24.62:33591
I20260808 20:12:23.952486 10799 heartbeater.cc:464] Registering TS with master...
I20260808 20:12:23.953353 10799 heartbeater.cc:511] Master 127.10.24.62:33591 requested a full tablet report, sending...
I20260808 20:12:23.955382 10389 ts_manager.cc:194] Registered new tserver with Master: 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:23.956477 10336 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.019393749s
I20260808 20:12:23.956827 10389 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40922
I20260808 20:12:23.960448 10336 test_util.cc:268] Using random seed: -1119871129
I20260808 20:12:24.018802 10389 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40924:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 20:12:24.024137 10389 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 20:12:24.152231 10738 tablet_service.cc:1467] Processing CreateTablet for tablet 7b193cc24b354319ab91439036f51881 (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260808 20:12:24.152400 10741 tablet_service.cc:1467] Processing CreateTablet for tablet 2a552acaffcd45268f344a1aa35cb132 (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260808 20:12:24.155143 10738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b193cc24b354319ab91439036f51881. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.155009 10739 tablet_service.cc:1467] Processing CreateTablet for tablet 10c2df6974bc48a19899f8ef6a1932dc (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260808 20:12:24.155100 10733 tablet_service.cc:1467] Processing CreateTablet for tablet e55ac233626f41e5b6ef9aa8f8204df8 (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260808 20:12:24.157760 10734 tablet_service.cc:1467] Processing CreateTablet for tablet 94846b349249412faaa65fb6ef97df95 (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260808 20:12:24.159408 10741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a552acaffcd45268f344a1aa35cb132. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.161700 10739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10c2df6974bc48a19899f8ef6a1932dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.162654 10732 tablet_service.cc:1467] Processing CreateTablet for tablet 4c8755efc8014e31a8f711fd137edef0 (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260808 20:12:24.164547 10733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e55ac233626f41e5b6ef9aa8f8204df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.165535 10732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c8755efc8014e31a8f711fd137edef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.167209 10734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94846b349249412faaa65fb6ef97df95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.172574 10730 tablet_service.cc:1467] Processing CreateTablet for tablet 3afafd38c69d4fa5b180f254a38ee912 (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260808 20:12:24.174573 10730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3afafd38c69d4fa5b180f254a38ee912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.186228 10729 tablet_service.cc:1467] Processing CreateTablet for tablet b99f754163dc41dbb325bba1ba45c63e (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260808 20:12:24.188457 10729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99f754163dc41dbb325bba1ba45c63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.194434 10728 tablet_service.cc:1467] Processing CreateTablet for tablet 00c98a0a1ac84a6aa4b7431b491e4e52 (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260808 20:12:24.196269 10728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00c98a0a1ac84a6aa4b7431b491e4e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.214228 10629 tablet_service.cc:1467] Processing CreateTablet for tablet 2a552acaffcd45268f344a1aa35cb132 (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260808 20:12:24.212929 10623 tablet_service.cc:1467] Processing CreateTablet for tablet 7b193cc24b354319ab91439036f51881 (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260808 20:12:24.223940 10629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a552acaffcd45268f344a1aa35cb132. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.221073 10620 tablet_service.cc:1467] Processing CreateTablet for tablet 94846b349249412faaa65fb6ef97df95 (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260808 20:12:24.215355 10624 tablet_service.cc:1467] Processing CreateTablet for tablet 10c2df6974bc48a19899f8ef6a1932dc (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260808 20:12:24.228261 10615 tablet_service.cc:1467] Processing CreateTablet for tablet 3afafd38c69d4fa5b180f254a38ee912 (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260808 20:12:24.229074 10618 tablet_service.cc:1467] Processing CreateTablet for tablet 4c8755efc8014e31a8f711fd137edef0 (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260808 20:12:24.214972 10619 tablet_service.cc:1467] Processing CreateTablet for tablet e55ac233626f41e5b6ef9aa8f8204df8 (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260808 20:12:24.231262 10624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10c2df6974bc48a19899f8ef6a1932dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.232398 10614 tablet_service.cc:1467] Processing CreateTablet for tablet b99f754163dc41dbb325bba1ba45c63e (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260808 20:12:24.235714 10615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3afafd38c69d4fa5b180f254a38ee912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.241909 10612 tablet_service.cc:1467] Processing CreateTablet for tablet 00c98a0a1ac84a6aa4b7431b491e4e52 (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260808 20:12:24.245110 10623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b193cc24b354319ab91439036f51881. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.260313 10620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94846b349249412faaa65fb6ef97df95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.267252 10612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00c98a0a1ac84a6aa4b7431b491e4e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.269377 10519 tablet_service.cc:1467] Processing CreateTablet for tablet 2a552acaffcd45268f344a1aa35cb132 (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260808 20:12:24.274912 10517 tablet_service.cc:1467] Processing CreateTablet for tablet 10c2df6974bc48a19899f8ef6a1932dc (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260808 20:12:24.283874 10619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e55ac233626f41e5b6ef9aa8f8204df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.292026 10519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a552acaffcd45268f344a1aa35cb132. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.291242 10516 tablet_service.cc:1467] Processing CreateTablet for tablet 7b193cc24b354319ab91439036f51881 (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260808 20:12:24.294679 10515 tablet_service.cc:1467] Processing CreateTablet for tablet 94846b349249412faaa65fb6ef97df95 (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260808 20:12:24.305380 10517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10c2df6974bc48a19899f8ef6a1932dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.305562 10614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99f754163dc41dbb325bba1ba45c63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.297170 10512 tablet_service.cc:1467] Processing CreateTablet for tablet e55ac233626f41e5b6ef9aa8f8204df8 (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
W20260808 20:12:24.313608 10455 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 20:12:24.315707 10516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b193cc24b354319ab91439036f51881. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.322808 10618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c8755efc8014e31a8f711fd137edef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.333154 10509 tablet_service.cc:1467] Processing CreateTablet for tablet b99f754163dc41dbb325bba1ba45c63e (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260808 20:12:24.335573 10512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e55ac233626f41e5b6ef9aa8f8204df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.336326 10515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94846b349249412faaa65fb6ef97df95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.332602 10508 tablet_service.cc:1467] Processing CreateTablet for tablet 00c98a0a1ac84a6aa4b7431b491e4e52 (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260808 20:12:24.334386 10511 tablet_service.cc:1467] Processing CreateTablet for tablet 4c8755efc8014e31a8f711fd137edef0 (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260808 20:12:24.334587 10510 tablet_service.cc:1467] Processing CreateTablet for tablet 3afafd38c69d4fa5b180f254a38ee912 (DEFAULT_TABLE table=test-workload [id=a83a65f58c334222ae57367d99407a34]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260808 20:12:24.379009 10509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99f754163dc41dbb325bba1ba45c63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.390182 10508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00c98a0a1ac84a6aa4b7431b491e4e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.392580 10511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c8755efc8014e31a8f711fd137edef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.396286 10826 tablet_bootstrap.cc:492] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe: Bootstrap starting.
I20260808 20:12:24.396845 10510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3afafd38c69d4fa5b180f254a38ee912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:24.430330 10828 tablet_bootstrap.cc:492] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263: Bootstrap starting.
I20260808 20:12:24.430778 10827 tablet_bootstrap.cc:492] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190: Bootstrap starting.
I20260808 20:12:24.449479 10826 tablet_bootstrap.cc:654] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:24.460812 10828 tablet_bootstrap.cc:654] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:24.467084 10827 tablet_bootstrap.cc:654] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:24.474041 10826 tablet_bootstrap.cc:492] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe: No bootstrap required, opened a new log
I20260808 20:12:24.474505 10826 ts_tablet_manager.cc:1397] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe: Time spent bootstrapping tablet: real 0.095s user 0.015s sys 0.007s
I20260808 20:12:24.477406 10826 raft_consensus.cc:348] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.478214 10826 raft_consensus.cc:374] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:24.478505 10826 raft_consensus.cc:729] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186c827b8b48447b94beb413077d76fe, State: Initialized, Role: FOLLOWER
I20260808 20:12:24.479173 10826 consensus_queue.cc:260] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.480973 10828 tablet_bootstrap.cc:492] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263: No bootstrap required, opened a new log
I20260808 20:12:24.481417 10828 ts_tablet_manager.cc:1397] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.008s
I20260808 20:12:24.484480 10828 raft_consensus.cc:348] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.486212 10828 raft_consensus.cc:374] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:24.486588 10828 raft_consensus.cc:729] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Initialized, Role: FOLLOWER
I20260808 20:12:24.487331 10828 consensus_queue.cc:260] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.487877 10827 tablet_bootstrap.cc:492] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190: No bootstrap required, opened a new log
I20260808 20:12:24.488410 10827 ts_tablet_manager.cc:1397] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190: Time spent bootstrapping tablet: real 0.084s user 0.012s sys 0.014s
I20260808 20:12:24.492316 10826 ts_tablet_manager.cc:1428] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe: Time spent starting tablet: real 0.017s user 0.010s sys 0.007s
I20260808 20:12:24.492588 10799 heartbeater.cc:503] Master 127.10.24.62:33591 was elected leader, sending a full tablet report...
I20260808 20:12:24.492594 10827 raft_consensus.cc:348] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } } peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } }
I20260808 20:12:24.493333 10826 tablet_bootstrap.cc:492] T 94846b349249412faaa65fb6ef97df95 P 186c827b8b48447b94beb413077d76fe: Bootstrap starting.
I20260808 20:12:24.493718 10827 raft_consensus.cc:374] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:24.494081 10827 raft_consensus.cc:729] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Initialized, Role: FOLLOWER
I20260808 20:12:24.494912 10827 consensus_queue.cc:260] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } } peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } }
I20260808 20:12:24.501085 10826 tablet_bootstrap.cc:654] T 94846b349249412faaa65fb6ef97df95 P 186c827b8b48447b94beb413077d76fe: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:24.502413 10565 heartbeater.cc:503] Master 127.10.24.62:33591 was elected leader, sending a full tablet report...
I20260808 20:12:24.510720 10686 heartbeater.cc:503] Master 127.10.24.62:33591 was elected leader, sending a full tablet report...
I20260808 20:12:24.514348 10828 ts_tablet_manager.cc:1428] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263: Time spent starting tablet: real 0.033s user 0.004s sys 0.006s
I20260808 20:12:24.515329 10828 tablet_bootstrap.cc:492] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263: Bootstrap starting.
I20260808 20:12:24.520150 10827 ts_tablet_manager.cc:1428] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190: Time spent starting tablet: real 0.031s user 0.000s sys 0.020s
I20260808 20:12:24.522192 10826 tablet_bootstrap.cc:492] T 94846b349249412faaa65fb6ef97df95 P 186c827b8b48447b94beb413077d76fe: No bootstrap required, opened a new log
I20260808 20:12:24.522625 10826 ts_tablet_manager.cc:1397] T 94846b349249412faaa65fb6ef97df95 P 186c827b8b48447b94beb413077d76fe: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.013s
I20260808 20:12:24.522966 10828 tablet_bootstrap.cc:654] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:24.525563 10826 raft_consensus.cc:348] T 94846b349249412faaa65fb6ef97df95 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.526302 10826 raft_consensus.cc:374] T 94846b349249412faaa65fb6ef97df95 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:24.526585 10826 raft_consensus.cc:729] T 94846b349249412faaa65fb6ef97df95 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186c827b8b48447b94beb413077d76fe, State: Initialized, Role: FOLLOWER
I20260808 20:12:24.527266 10826 consensus_queue.cc:260] T 94846b349249412faaa65fb6ef97df95 P 186c827b8b48447b94beb413077d76fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.530037 10826 ts_tablet_manager.cc:1428] T 94846b349249412faaa65fb6ef97df95 P 186c827b8b48447b94beb413077d76fe: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 20:12:24.530921 10826 tablet_bootstrap.cc:492] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe: Bootstrap starting.
I20260808 20:12:24.533248 10827 tablet_bootstrap.cc:492] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190: Bootstrap starting.
I20260808 20:12:24.535106 10828 tablet_bootstrap.cc:492] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263: No bootstrap required, opened a new log
I20260808 20:12:24.535564 10828 ts_tablet_manager.cc:1397] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.010s
I20260808 20:12:24.537122 10826 tablet_bootstrap.cc:654] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:24.538646 10828 raft_consensus.cc:348] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.539417 10828 raft_consensus.cc:374] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:24.539714 10828 raft_consensus.cc:729] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Initialized, Role: FOLLOWER
I20260808 20:12:24.542205 10828 consensus_queue.cc:260] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.544548 10828 ts_tablet_manager.cc:1428] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263: Time spent starting tablet: real 0.009s user 0.005s sys 0.002s
I20260808 20:12:24.545449 10828 tablet_bootstrap.cc:492] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263: Bootstrap starting.
I20260808 20:12:24.547158 10827 tablet_bootstrap.cc:654] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:24.551496 10828 tablet_bootstrap.cc:654] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:24.553702 10826 tablet_bootstrap.cc:492] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe: No bootstrap required, opened a new log
I20260808 20:12:24.554157 10826 ts_tablet_manager.cc:1397] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.011s
I20260808 20:12:24.559618 10827 tablet_bootstrap.cc:492] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190: No bootstrap required, opened a new log
I20260808 20:12:24.560086 10827 ts_tablet_manager.cc:1397] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.008s
I20260808 20:12:24.563681 10827 raft_consensus.cc:348] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.564502 10827 raft_consensus.cc:374] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:24.564848 10827 raft_consensus.cc:729] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Initialized, Role: FOLLOWER
W20260808 20:12:24.562152 10566 tablet.cc:2369] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 20:12:24.565645 10827 consensus_queue.cc:260] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.569223 10827 ts_tablet_manager.cc:1428] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260808 20:12:24.569274 10828 tablet_bootstrap.cc:492] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263: No bootstrap required, opened a new log
I20260808 20:12:24.569869 10828 ts_tablet_manager.cc:1397] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.013s
I20260808 20:12:24.571696 10826 raft_consensus.cc:348] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } } peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } }
I20260808 20:12:24.572399 10826 raft_consensus.cc:374] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:24.572292 10828 raft_consensus.cc:348] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } } peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } }
I20260808 20:12:24.572759 10826 raft_consensus.cc:729] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186c827b8b48447b94beb413077d76fe, State: Initialized, Role: FOLLOWER
I20260808 20:12:24.573009 10828 raft_consensus.cc:374] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:24.573310 10828 raft_consensus.cc:729] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Initialized, Role: FOLLOWER
I20260808 20:12:24.574043 10835 raft_consensus.cc:482] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:24.573840 10826 consensus_queue.cc:260] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } } peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } }
I20260808 20:12:24.574483 10835 raft_consensus.cc:504] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.575327 10827 tablet_bootstrap.cc:492] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190: Bootstrap starting.
I20260808 20:12:24.578395 10835 leader_election.cc:288] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:24.597661 10833 raft_consensus.cc:482] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:24.577610 10828 consensus_queue.cc:260] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } } peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } }
I20260808 20:12:24.598642 10833 raft_consensus.cc:504] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } } peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } }
I20260808 20:12:24.602304 10828 ts_tablet_manager.cc:1428] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263: Time spent starting tablet: real 0.032s user 0.002s sys 0.004s
I20260808 20:12:24.603929 10826 ts_tablet_manager.cc:1428] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe: Time spent starting tablet: real 0.049s user 0.004s sys 0.003s
I20260808 20:12:24.606129 10833 leader_election.cc:288] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407)
I20260808 20:12:24.610536 10826 tablet_bootstrap.cc:492] T e55ac233626f41e5b6ef9aa8f8204df8 P 186c827b8b48447b94beb413077d76fe: Bootstrap starting.
I20260808 20:12:24.610018 10828 tablet_bootstrap.cc:492] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263: Bootstrap starting.
I20260808 20:12:24.612145 10827 tablet_bootstrap.cc:654] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:24.642824 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94846b349249412faaa65fb6ef97df95" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186c827b8b48447b94beb413077d76fe" is_pre_election: true
I20260808 20:12:24.645424 10761 raft_consensus.cc:2454] T 94846b349249412faaa65fb6ef97df95 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7858bb58840c4165b68c2f114ae20263 in term 0.
I20260808 20:12:24.647470 10480 leader_election.cc:302] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, 7858bb58840c4165b68c2f114ae20263; no voters:
I20260808 20:12:24.649242 10835 raft_consensus.cc:2781] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:12:24.649718 10835 raft_consensus.cc:482] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:12:24.650072 10835 raft_consensus.cc:3037] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:24.653739 10828 tablet_bootstrap.cc:654] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:24.657300 10826 tablet_bootstrap.cc:654] T e55ac233626f41e5b6ef9aa8f8204df8 P 186c827b8b48447b94beb413077d76fe: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:24.657590 10835 raft_consensus.cc:504] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.663539 10827 tablet_bootstrap.cc:492] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190: No bootstrap required, opened a new log
I20260808 20:12:24.664480 10827 ts_tablet_manager.cc:1397] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190: Time spent bootstrapping tablet: real 0.089s user 0.008s sys 0.009s
I20260808 20:12:24.664976 10645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94846b349249412faaa65fb6ef97df95" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" is_pre_election: true
I20260808 20:12:24.665511 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94846b349249412faaa65fb6ef97df95" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
W20260808 20:12:24.667438 10481 leader_election.cc:341] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:12:24.667415 10827 raft_consensus.cc:348] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.668295 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94846b349249412faaa65fb6ef97df95" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186c827b8b48447b94beb413077d76fe"
W20260808 20:12:24.668619 10481 leader_election.cc:341] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:12:24.668871 10761 raft_consensus.cc:3037] T 94846b349249412faaa65fb6ef97df95 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:24.668270 10827 raft_consensus.cc:374] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:24.672330 10827 raft_consensus.cc:729] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Initialized, Role: FOLLOWER
I20260808 20:12:24.673013 10827 consensus_queue.cc:260] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.675519 10827 ts_tablet_manager.cc:1428] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190: Time spent starting tablet: real 0.011s user 0.004s sys 0.004s
I20260808 20:12:24.676057 10761 raft_consensus.cc:2454] T 94846b349249412faaa65fb6ef97df95 P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7858bb58840c4165b68c2f114ae20263 in term 1.
I20260808 20:12:24.676437 10827 tablet_bootstrap.cc:492] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190: Bootstrap starting.
I20260808 20:12:24.677325 10480 leader_election.cc:302] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, 7858bb58840c4165b68c2f114ae20263; no voters: ff9f26e2cbef4d1fbf9ca975e1959190
I20260808 20:12:24.687426 10835 leader_election.cc:288] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 election: Requested vote from peers ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:24.687968 10835 raft_consensus.cc:2781] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:12:24.691504 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a552acaffcd45268f344a1aa35cb132" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" is_pre_election: true
I20260808 20:12:24.692147 10644 raft_consensus.cc:2454] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186c827b8b48447b94beb413077d76fe in term 0.
I20260808 20:12:24.692494 10827 tablet_bootstrap.cc:654] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:24.692672 10828 tablet_bootstrap.cc:492] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263: No bootstrap required, opened a new log
I20260808 20:12:24.693354 10828 ts_tablet_manager.cc:1397] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263: Time spent bootstrapping tablet: real 0.089s user 0.020s sys 0.009s
I20260808 20:12:24.693576 10704 leader_election.cc:302] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, ff9f26e2cbef4d1fbf9ca975e1959190; no voters:
I20260808 20:12:24.694391 10833 raft_consensus.cc:2781] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:12:24.694741 10833 raft_consensus.cc:482] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:12:24.695106 10833 raft_consensus.cc:3037] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:24.688390 10835 raft_consensus.cc:686] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [term 1 LEADER]: Becoming Leader. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Running, Role: LEADER
I20260808 20:12:24.706913 10826 tablet_bootstrap.cc:492] T e55ac233626f41e5b6ef9aa8f8204df8 P 186c827b8b48447b94beb413077d76fe: No bootstrap required, opened a new log
I20260808 20:12:24.707377 10826 ts_tablet_manager.cc:1397] T e55ac233626f41e5b6ef9aa8f8204df8 P 186c827b8b48447b94beb413077d76fe: Time spent bootstrapping tablet: real 0.103s user 0.004s sys 0.023s
I20260808 20:12:24.707672 10828 raft_consensus.cc:348] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.708391 10828 raft_consensus.cc:374] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:24.708666 10828 raft_consensus.cc:729] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Initialized, Role: FOLLOWER
I20260808 20:12:24.709337 10828 consensus_queue.cc:260] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.710109 10826 raft_consensus.cc:348] T e55ac233626f41e5b6ef9aa8f8204df8 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.710867 10826 raft_consensus.cc:374] T e55ac233626f41e5b6ef9aa8f8204df8 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:24.711158 10826 raft_consensus.cc:729] T e55ac233626f41e5b6ef9aa8f8204df8 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186c827b8b48447b94beb413077d76fe, State: Initialized, Role: FOLLOWER
I20260808 20:12:24.711537 10828 ts_tablet_manager.cc:1428] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260808 20:12:24.711795 10826 consensus_queue.cc:260] T e55ac233626f41e5b6ef9aa8f8204df8 P 186c827b8b48447b94beb413077d76fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.712468 10828 tablet_bootstrap.cc:492] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263: Bootstrap starting.
I20260808 20:12:24.713550 10532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a552acaffcd45268f344a1aa35cb132" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7858bb58840c4165b68c2f114ae20263" is_pre_election: true
I20260808 20:12:24.714205 10532 raft_consensus.cc:2454] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186c827b8b48447b94beb413077d76fe in term 0.
I20260808 20:12:24.716879 10833 raft_consensus.cc:504] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } } peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } }
I20260808 20:12:24.718742 10833 leader_election.cc:288] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 1 election: Requested vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407)
I20260808 20:12:24.719110 10828 tablet_bootstrap.cc:654] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:24.719872 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a552acaffcd45268f344a1aa35cb132" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
I20260808 20:12:24.720443 10644 raft_consensus.cc:3037] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:24.720373 10835 consensus_queue.cc:237] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.725575 10532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a552acaffcd45268f344a1aa35cb132" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7858bb58840c4165b68c2f114ae20263"
I20260808 20:12:24.726095 10532 raft_consensus.cc:3037] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:24.728335 10644 raft_consensus.cc:2454] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186c827b8b48447b94beb413077d76fe in term 1.
I20260808 20:12:24.733291 10827 tablet_bootstrap.cc:492] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190: No bootstrap required, opened a new log
I20260808 20:12:24.733757 10827 ts_tablet_manager.cc:1397] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190: Time spent bootstrapping tablet: real 0.057s user 0.017s sys 0.008s
I20260808 20:12:24.740747 10704 leader_election.cc:302] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, ff9f26e2cbef4d1fbf9ca975e1959190; no voters:
I20260808 20:12:24.741598 10532 raft_consensus.cc:2454] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186c827b8b48447b94beb413077d76fe in term 1.
I20260808 20:12:24.744146 10828 tablet_bootstrap.cc:492] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263: No bootstrap required, opened a new log
I20260808 20:12:24.744565 10828 ts_tablet_manager.cc:1397] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.009s
I20260808 20:12:24.744616 10827 raft_consensus.cc:348] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.745188 10857 raft_consensus.cc:2781] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:12:24.745643 10827 raft_consensus.cc:374] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:24.746017 10827 raft_consensus.cc:729] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Initialized, Role: FOLLOWER
I20260808 20:12:24.746153 10857 raft_consensus.cc:686] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [term 1 LEADER]: Becoming Leader. State: Replica: 186c827b8b48447b94beb413077d76fe, State: Running, Role: LEADER
I20260808 20:12:24.746901 10827 consensus_queue.cc:260] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.748152 10828 raft_consensus.cc:348] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.747577 10826 ts_tablet_manager.cc:1428] T e55ac233626f41e5b6ef9aa8f8204df8 P 186c827b8b48447b94beb413077d76fe: Time spent starting tablet: real 0.040s user 0.012s sys 0.005s
I20260808 20:12:24.749118 10828 raft_consensus.cc:374] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:24.749728 10828 raft_consensus.cc:729] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Initialized, Role: FOLLOWER
I20260808 20:12:24.748443 10857 consensus_queue.cc:237] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } } peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } }
I20260808 20:12:24.750739 10828 consensus_queue.cc:260] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.753082 10827 ts_tablet_manager.cc:1428] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190: Time spent starting tablet: real 0.019s user 0.006s sys 0.003s
I20260808 20:12:24.753541 10826 tablet_bootstrap.cc:492] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe: Bootstrap starting.
I20260808 20:12:24.757495 10828 ts_tablet_manager.cc:1428] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20260808 20:12:24.763983 10384 catalog_manager.cc:5495] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7858bb58840c4165b68c2f114ae20263 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "7858bb58840c4165b68c2f114ae20263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:24.768736 10827 tablet_bootstrap.cc:492] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190: Bootstrap starting.
I20260808 20:12:24.770593 10389 catalog_manager.cc:5495] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 186c827b8b48447b94beb413077d76fe (127.10.24.3). New cstate: current_term: 1 leader_uuid: "186c827b8b48447b94beb413077d76fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:24.772167 10828 tablet_bootstrap.cc:492] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263: Bootstrap starting.
I20260808 20:12:24.777055 10826 tablet_bootstrap.cc:654] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:24.784785 10828 tablet_bootstrap.cc:654] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:24.800125 10827 tablet_bootstrap.cc:654] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:24.808192 10826 tablet_bootstrap.cc:492] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe: No bootstrap required, opened a new log
I20260808 20:12:24.808732 10826 ts_tablet_manager.cc:1397] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe: Time spent bootstrapping tablet: real 0.055s user 0.013s sys 0.024s
I20260808 20:12:24.808733 10828 tablet_bootstrap.cc:492] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263: No bootstrap required, opened a new log
I20260808 20:12:24.809491 10828 ts_tablet_manager.cc:1397] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263: Time spent bootstrapping tablet: real 0.038s user 0.011s sys 0.013s
I20260808 20:12:24.812470 10828 raft_consensus.cc:348] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.813316 10828 raft_consensus.cc:374] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:24.813606 10828 raft_consensus.cc:729] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Initialized, Role: FOLLOWER
I20260808 20:12:24.812449 10826 raft_consensus.cc:348] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.814952 10826 raft_consensus.cc:374] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:24.814421 10828 consensus_queue.cc:260] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.815901 10826 raft_consensus.cc:729] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186c827b8b48447b94beb413077d76fe, State: Initialized, Role: FOLLOWER
I20260808 20:12:24.819664 10827 tablet_bootstrap.cc:492] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190: No bootstrap required, opened a new log
I20260808 20:12:24.822026 10827 ts_tablet_manager.cc:1397] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190: Time spent bootstrapping tablet: real 0.053s user 0.008s sys 0.006s
I20260808 20:12:24.821560 10828 ts_tablet_manager.cc:1428] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260808 20:12:24.824019 10826 consensus_queue.cc:260] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.826536 10826 ts_tablet_manager.cc:1428] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260808 20:12:24.826592 10827 raft_consensus.cc:348] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.827391 10826 tablet_bootstrap.cc:492] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe: Bootstrap starting.
I20260808 20:12:24.827440 10827 raft_consensus.cc:374] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:24.827894 10827 raft_consensus.cc:729] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Initialized, Role: FOLLOWER
I20260808 20:12:24.828789 10827 consensus_queue.cc:260] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.831373 10828 tablet_bootstrap.cc:492] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263: Bootstrap starting.
I20260808 20:12:24.832103 10857 raft_consensus.cc:482] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:24.833326 10857 raft_consensus.cc:504] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.835595 10857 leader_election.cc:288] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407)
I20260808 20:12:24.838748 10827 ts_tablet_manager.cc:1428] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190: Time spent starting tablet: real 0.016s user 0.009s sys 0.000s
I20260808 20:12:24.838657 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c2df6974bc48a19899f8ef6a1932dc" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" is_pre_election: true
I20260808 20:12:24.839030 10532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c2df6974bc48a19899f8ef6a1932dc" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7858bb58840c4165b68c2f114ae20263" is_pre_election: true
I20260808 20:12:24.839318 10644 raft_consensus.cc:2454] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186c827b8b48447b94beb413077d76fe in term 0.
I20260808 20:12:24.839627 10827 tablet_bootstrap.cc:492] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190: Bootstrap starting.
I20260808 20:12:24.839730 10532 raft_consensus.cc:2454] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186c827b8b48447b94beb413077d76fe in term 0.
I20260808 20:12:24.841033 10705 leader_election.cc:302] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, 7858bb58840c4165b68c2f114ae20263; no voters:
I20260808 20:12:24.841792 10857 raft_consensus.cc:2781] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:12:24.842114 10857 raft_consensus.cc:482] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:12:24.842437 10857 raft_consensus.cc:3037] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:24.841506 10826 tablet_bootstrap.cc:654] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:24.852962 10857 raft_consensus.cc:504] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.857079 10827 tablet_bootstrap.cc:654] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:24.858094 10532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c2df6974bc48a19899f8ef6a1932dc" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7858bb58840c4165b68c2f114ae20263"
I20260808 20:12:24.858319 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c2df6974bc48a19899f8ef6a1932dc" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
I20260808 20:12:24.858146 10857 leader_election.cc:288] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 1 election: Requested vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407)
I20260808 20:12:24.858757 10532 raft_consensus.cc:3037] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:24.858781 10644 raft_consensus.cc:3037] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:24.862442 10826 tablet_bootstrap.cc:492] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe: No bootstrap required, opened a new log
I20260808 20:12:24.862938 10826 ts_tablet_manager.cc:1397] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe: Time spent bootstrapping tablet: real 0.036s user 0.016s sys 0.006s
I20260808 20:12:24.865456 10532 raft_consensus.cc:2454] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186c827b8b48447b94beb413077d76fe in term 1.
I20260808 20:12:24.865747 10828 tablet_bootstrap.cc:654] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:24.866580 10705 leader_election.cc:302] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, 7858bb58840c4165b68c2f114ae20263; no voters:
I20260808 20:12:24.866779 10644 raft_consensus.cc:2454] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186c827b8b48447b94beb413077d76fe in term 1.
I20260808 20:12:24.865562 10826 raft_consensus.cc:348] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.867509 10826 raft_consensus.cc:374] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:24.867821 10826 raft_consensus.cc:729] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186c827b8b48447b94beb413077d76fe, State: Initialized, Role: FOLLOWER
I20260808 20:12:24.867887 10857 raft_consensus.cc:2781] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:12:24.868659 10857 raft_consensus.cc:686] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 1 LEADER]: Becoming Leader. State: Replica: 186c827b8b48447b94beb413077d76fe, State: Running, Role: LEADER
I20260808 20:12:24.868602 10826 consensus_queue.cc:260] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.869554 10857 consensus_queue.cc:237] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.871119 10826 ts_tablet_manager.cc:1428] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260808 20:12:24.872098 10826 tablet_bootstrap.cc:492] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe: Bootstrap starting.
I20260808 20:12:24.882869 10835 raft_consensus.cc:482] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:24.883368 10835 raft_consensus.cc:504] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.886497 10835 leader_election.cc:288] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:24.887048 10826 tablet_bootstrap.cc:654] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:24.887526 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b193cc24b354319ab91439036f51881" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186c827b8b48447b94beb413077d76fe" is_pre_election: true
I20260808 20:12:24.885950 10389 catalog_manager.cc:5495] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 186c827b8b48447b94beb413077d76fe (127.10.24.3). New cstate: current_term: 1 leader_uuid: "186c827b8b48447b94beb413077d76fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: HEALTHY } } }
I20260808 20:12:24.888135 10761 raft_consensus.cc:2454] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7858bb58840c4165b68c2f114ae20263 in term 0.
I20260808 20:12:24.889921 10480 leader_election.cc:302] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, 7858bb58840c4165b68c2f114ae20263; no voters:
I20260808 20:12:24.888449 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b193cc24b354319ab91439036f51881" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" is_pre_election: true
I20260808 20:12:24.890689 10835 raft_consensus.cc:2781] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:12:24.890887 10644 raft_consensus.cc:2454] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7858bb58840c4165b68c2f114ae20263 in term 0.
I20260808 20:12:24.891103 10835 raft_consensus.cc:482] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:12:24.891554 10835 raft_consensus.cc:3037] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:24.898066 10835 raft_consensus.cc:504] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.900704 10835 leader_election.cc:288] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 election: Requested vote from peers ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:24.902324 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b193cc24b354319ab91439036f51881" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
I20260808 20:12:24.902359 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b193cc24b354319ab91439036f51881" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186c827b8b48447b94beb413077d76fe"
I20260808 20:12:24.902935 10644 raft_consensus.cc:3037] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:24.903046 10761 raft_consensus.cc:3037] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:24.906173 10837 raft_consensus.cc:482] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:24.906678 10837 raft_consensus.cc:504] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.909714 10837 leader_election.cc:288] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:24.914424 10644 raft_consensus.cc:2454] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7858bb58840c4165b68c2f114ae20263 in term 1.
I20260808 20:12:24.915467 10481 leader_election.cc:302] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7858bb58840c4165b68c2f114ae20263, ff9f26e2cbef4d1fbf9ca975e1959190; no voters:
I20260808 20:12:24.916297 10835 raft_consensus.cc:2781] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:12:24.916857 10835 raft_consensus.cc:686] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [term 1 LEADER]: Becoming Leader. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Running, Role: LEADER
I20260808 20:12:24.917698 10835 consensus_queue.cc:237] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.922194 10761 raft_consensus.cc:2454] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7858bb58840c4165b68c2f114ae20263 in term 1.
I20260808 20:12:24.925873 10855 raft_consensus.cc:482] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:24.926322 10855 raft_consensus.cc:504] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.928812 10855 leader_election.cc:288] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:24.931635 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8755efc8014e31a8f711fd137edef0" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186c827b8b48447b94beb413077d76fe" is_pre_election: true
I20260808 20:12:24.931805 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8755efc8014e31a8f711fd137edef0" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" is_pre_election: true
I20260808 20:12:24.932359 10761 raft_consensus.cc:2454] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7858bb58840c4165b68c2f114ae20263 in term 0.
I20260808 20:12:24.933105 10827 tablet_bootstrap.cc:492] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190: No bootstrap required, opened a new log
I20260808 20:12:24.933573 10827 ts_tablet_manager.cc:1397] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190: Time spent bootstrapping tablet: real 0.094s user 0.004s sys 0.014s
I20260808 20:12:24.933418 10480 leader_election.cc:302] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, 7858bb58840c4165b68c2f114ae20263; no voters:
W20260808 20:12:24.934346 10481 leader_election.cc:341] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:12:24.935348 10855 raft_consensus.cc:2781] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:12:24.935680 10855 raft_consensus.cc:482] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:12:24.935976 10855 raft_consensus.cc:3037] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:24.936632 10827 raft_consensus.cc:348] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.940690 10827 raft_consensus.cc:374] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:24.941008 10827 raft_consensus.cc:729] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Initialized, Role: FOLLOWER
I20260808 20:12:24.941668 10827 consensus_queue.cc:260] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.942891 10855 raft_consensus.cc:504] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.944156 10827 ts_tablet_manager.cc:1428] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260808 20:12:24.945509 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8755efc8014e31a8f711fd137edef0" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
I20260808 20:12:24.946651 10828 tablet_bootstrap.cc:492] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263: No bootstrap required, opened a new log
I20260808 20:12:24.947067 10855 leader_election.cc:288] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 election: Requested vote from peers ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:24.946681 10760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8755efc8014e31a8f711fd137edef0" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186c827b8b48447b94beb413077d76fe"
I20260808 20:12:24.947794 10760 raft_consensus.cc:3037] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:24.947273 10828 ts_tablet_manager.cc:1397] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263: Time spent bootstrapping tablet: real 0.116s user 0.015s sys 0.052s
W20260808 20:12:24.946696 10481 leader_election.cc:341] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:12:24.953380 10828 raft_consensus.cc:348] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.953125 10389 catalog_manager.cc:5495] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7858bb58840c4165b68c2f114ae20263 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "7858bb58840c4165b68c2f114ae20263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:24.954069 10828 raft_consensus.cc:374] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:24.954372 10828 raft_consensus.cc:729] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Initialized, Role: FOLLOWER
I20260808 20:12:24.954993 10828 consensus_queue.cc:260] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.955464 10826 tablet_bootstrap.cc:492] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe: No bootstrap required, opened a new log
I20260808 20:12:24.957955 10826 ts_tablet_manager.cc:1397] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe: Time spent bootstrapping tablet: real 0.086s user 0.004s sys 0.015s
I20260808 20:12:24.956804 10827 tablet_bootstrap.cc:492] T 00c98a0a1ac84a6aa4b7431b491e4e52 P ff9f26e2cbef4d1fbf9ca975e1959190: Bootstrap starting.
I20260808 20:12:24.965649 10828 ts_tablet_manager.cc:1428] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20260808 20:12:24.957239 10760 raft_consensus.cc:2454] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7858bb58840c4165b68c2f114ae20263 in term 1.
I20260808 20:12:24.967427 10480 leader_election.cc:302] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, 7858bb58840c4165b68c2f114ae20263; no voters: ff9f26e2cbef4d1fbf9ca975e1959190
I20260808 20:12:24.967797 10835 raft_consensus.cc:482] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:24.968310 10855 raft_consensus.cc:2781] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:12:24.970780 10835 raft_consensus.cc:504] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.971004 10828 tablet_bootstrap.cc:492] T b99f754163dc41dbb325bba1ba45c63e P 7858bb58840c4165b68c2f114ae20263: Bootstrap starting.
I20260808 20:12:24.976099 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55ac233626f41e5b6ef9aa8f8204df8" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" is_pre_election: true
I20260808 20:12:24.983302 10644 raft_consensus.cc:2454] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7858bb58840c4165b68c2f114ae20263 in term 0.
I20260808 20:12:24.984674 10481 leader_election.cc:302] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7858bb58840c4165b68c2f114ae20263, ff9f26e2cbef4d1fbf9ca975e1959190; no voters:
I20260808 20:12:24.960721 10826 raft_consensus.cc:348] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.977262 10827 tablet_bootstrap.cc:654] T 00c98a0a1ac84a6aa4b7431b491e4e52 P ff9f26e2cbef4d1fbf9ca975e1959190: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:24.989332 10826 raft_consensus.cc:374] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:24.990016 10826 raft_consensus.cc:729] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186c827b8b48447b94beb413077d76fe, State: Initialized, Role: FOLLOWER
I20260808 20:12:24.991379 10826 consensus_queue.cc:260] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:24.978857 10760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55ac233626f41e5b6ef9aa8f8204df8" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186c827b8b48447b94beb413077d76fe" is_pre_election: true
I20260808 20:12:24.979293 10835 leader_election.cc:288] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:24.995710 10884 raft_consensus.cc:2781] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:12:24.995831 10532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55ac233626f41e5b6ef9aa8f8204df8" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7858bb58840c4165b68c2f114ae20263" is_pre_election: true
I20260808 20:12:24.994930 10855 raft_consensus.cc:686] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [term 1 LEADER]: Becoming Leader. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Running, Role: LEADER
I20260808 20:12:24.994285 10760 raft_consensus.cc:2454] T e55ac233626f41e5b6ef9aa8f8204df8 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7858bb58840c4165b68c2f114ae20263 in term 0.
I20260808 20:12:24.997855 10884 raft_consensus.cc:482] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:12:24.998476 10884 raft_consensus.cc:3037] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:24.999199 10828 tablet_bootstrap.cc:654] T b99f754163dc41dbb325bba1ba45c63e P 7858bb58840c4165b68c2f114ae20263: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:25.001348 10826 ts_tablet_manager.cc:1428] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe: Time spent starting tablet: real 0.043s user 0.001s sys 0.006s
I20260808 20:12:25.001541 10760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55ac233626f41e5b6ef9aa8f8204df8" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186c827b8b48447b94beb413077d76fe" is_pre_election: true
I20260808 20:12:25.002233 10760 raft_consensus.cc:2454] T e55ac233626f41e5b6ef9aa8f8204df8 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff9f26e2cbef4d1fbf9ca975e1959190 in term 0.
I20260808 20:12:25.002605 10826 tablet_bootstrap.cc:492] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe: Bootstrap starting.
I20260808 20:12:25.003435 10586 leader_election.cc:302] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, ff9f26e2cbef4d1fbf9ca975e1959190; no voters:
I20260808 20:12:25.004451 10837 raft_consensus.cc:2781] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:12:25.004820 10837 raft_consensus.cc:482] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:12:25.005196 10837 raft_consensus.cc:3037] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:25.004047 10827 tablet_bootstrap.cc:492] T 00c98a0a1ac84a6aa4b7431b491e4e52 P ff9f26e2cbef4d1fbf9ca975e1959190: No bootstrap required, opened a new log
I20260808 20:12:25.004469 10855 consensus_queue.cc:237] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.007864 10827 ts_tablet_manager.cc:1397] T 00c98a0a1ac84a6aa4b7431b491e4e52 P ff9f26e2cbef4d1fbf9ca975e1959190: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.013s
I20260808 20:12:25.013484 10837 raft_consensus.cc:504] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.015301 10837 leader_election.cc:288] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 1 election: Requested vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:25.016288 10529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55ac233626f41e5b6ef9aa8f8204df8" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7858bb58840c4165b68c2f114ae20263"
I20260808 20:12:25.018045 10884 raft_consensus.cc:504] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.019797 10760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55ac233626f41e5b6ef9aa8f8204df8" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186c827b8b48447b94beb413077d76fe"
I20260808 20:12:25.020514 10760 raft_consensus.cc:3037] T e55ac233626f41e5b6ef9aa8f8204df8 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:25.021509 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55ac233626f41e5b6ef9aa8f8204df8" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186c827b8b48447b94beb413077d76fe"
I20260808 20:12:25.021899 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55ac233626f41e5b6ef9aa8f8204df8" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
I20260808 20:12:25.022702 10532 raft_consensus.cc:2379] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff9f26e2cbef4d1fbf9ca975e1959190 in current term 1: Already voted for candidate 7858bb58840c4165b68c2f114ae20263 in this term.
I20260808 20:12:25.023682 10837 raft_consensus.cc:482] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:25.024097 10837 raft_consensus.cc:504] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.025748 10644 raft_consensus.cc:2379] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7858bb58840c4165b68c2f114ae20263 in current term 1: Already voted for candidate ff9f26e2cbef4d1fbf9ca975e1959190 in this term.
I20260808 20:12:25.025906 10837 leader_election.cc:288] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:25.026336 10884 leader_election.cc:288] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 election: Requested vote from peers ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:25.026584 10532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94846b349249412faaa65fb6ef97df95" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7858bb58840c4165b68c2f114ae20263" is_pre_election: true
I20260808 20:12:25.027104 10481 leader_election.cc:302] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7858bb58840c4165b68c2f114ae20263; no voters: 186c827b8b48447b94beb413077d76fe, ff9f26e2cbef4d1fbf9ca975e1959190
I20260808 20:12:25.027868 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94846b349249412faaa65fb6ef97df95" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186c827b8b48447b94beb413077d76fe" is_pre_election: true
I20260808 20:12:25.029866 10761 raft_consensus.cc:2379] T 94846b349249412faaa65fb6ef97df95 P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff9f26e2cbef4d1fbf9ca975e1959190 in current term 1: Already voted for candidate 7858bb58840c4165b68c2f114ae20263 in this term.
I20260808 20:12:25.028836 10827 raft_consensus.cc:348] T 00c98a0a1ac84a6aa4b7431b491e4e52 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.031330 10586 leader_election.cc:302] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff9f26e2cbef4d1fbf9ca975e1959190; no voters: 186c827b8b48447b94beb413077d76fe, 7858bb58840c4165b68c2f114ae20263
I20260808 20:12:25.031971 10827 raft_consensus.cc:374] T 00c98a0a1ac84a6aa4b7431b491e4e52 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:25.032354 10827 raft_consensus.cc:729] T 00c98a0a1ac84a6aa4b7431b491e4e52 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Initialized, Role: FOLLOWER
I20260808 20:12:25.033382 10826 tablet_bootstrap.cc:654] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:25.033063 10827 consensus_queue.cc:260] T 00c98a0a1ac84a6aa4b7431b491e4e52 P ff9f26e2cbef4d1fbf9ca975e1959190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.034078 10837 raft_consensus.cc:3037] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:25.036948 10384 catalog_manager.cc:5495] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7858bb58840c4165b68c2f114ae20263 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "7858bb58840c4165b68c2f114ae20263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:25.030989 10884 raft_consensus.cc:2726] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 20:12:25.044126 10855 raft_consensus.cc:482] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:25.047262 10855 raft_consensus.cc:504] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.047916 10837 raft_consensus.cc:2726] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:12:25.051797 10855 leader_election.cc:288] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:25.052661 10827 ts_tablet_manager.cc:1428] T 00c98a0a1ac84a6aa4b7431b491e4e52 P ff9f26e2cbef4d1fbf9ca975e1959190: Time spent starting tablet: real 0.040s user 0.014s sys 0.005s
I20260808 20:12:25.053392 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c98a0a1ac84a6aa4b7431b491e4e52" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186c827b8b48447b94beb413077d76fe" is_pre_election: true
W20260808 20:12:25.054936 10480 leader_election.cc:341] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:12:25.055115 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c98a0a1ac84a6aa4b7431b491e4e52" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" is_pre_election: true
I20260808 20:12:25.055699 10644 raft_consensus.cc:2454] T 00c98a0a1ac84a6aa4b7431b491e4e52 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7858bb58840c4165b68c2f114ae20263 in term 0.
I20260808 20:12:25.057431 10481 leader_election.cc:302] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7858bb58840c4165b68c2f114ae20263, ff9f26e2cbef4d1fbf9ca975e1959190; no voters: 186c827b8b48447b94beb413077d76fe
I20260808 20:12:25.058239 10827 tablet_bootstrap.cc:492] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190: Bootstrap starting.
I20260808 20:12:25.059285 10760 raft_consensus.cc:2454] T e55ac233626f41e5b6ef9aa8f8204df8 P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff9f26e2cbef4d1fbf9ca975e1959190 in term 1.
I20260808 20:12:25.060266 10586 leader_election.cc:302] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, ff9f26e2cbef4d1fbf9ca975e1959190; no voters: 7858bb58840c4165b68c2f114ae20263
I20260808 20:12:25.060456 10855 raft_consensus.cc:2781] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:12:25.057821 10828 tablet_bootstrap.cc:492] T b99f754163dc41dbb325bba1ba45c63e P 7858bb58840c4165b68c2f114ae20263: No bootstrap required, opened a new log
I20260808 20:12:25.060810 10855 raft_consensus.cc:482] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:12:25.061570 10855 raft_consensus.cc:3037] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:25.061657 10828 ts_tablet_manager.cc:1397] T b99f754163dc41dbb325bba1ba45c63e P 7858bb58840c4165b68c2f114ae20263: Time spent bootstrapping tablet: real 0.095s user 0.007s sys 0.018s
I20260808 20:12:25.064018 10826 tablet_bootstrap.cc:492] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe: No bootstrap required, opened a new log
I20260808 20:12:25.064639 10826 ts_tablet_manager.cc:1397] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe: Time spent bootstrapping tablet: real 0.062s user 0.006s sys 0.010s
I20260808 20:12:25.066505 10827 tablet_bootstrap.cc:654] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:25.066601 10828 raft_consensus.cc:348] T b99f754163dc41dbb325bba1ba45c63e P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.067281 10828 raft_consensus.cc:374] T b99f754163dc41dbb325bba1ba45c63e P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:25.067567 10828 raft_consensus.cc:729] T b99f754163dc41dbb325bba1ba45c63e P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Initialized, Role: FOLLOWER
I20260808 20:12:25.067777 10826 raft_consensus.cc:348] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.068413 10826 raft_consensus.cc:374] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:25.068192 10828 consensus_queue.cc:260] T b99f754163dc41dbb325bba1ba45c63e P 7858bb58840c4165b68c2f114ae20263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.068660 10826 raft_consensus.cc:729] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186c827b8b48447b94beb413077d76fe, State: Initialized, Role: FOLLOWER
I20260808 20:12:25.061410 10837 raft_consensus.cc:2781] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:12:25.069288 10826 consensus_queue.cc:260] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.069718 10855 raft_consensus.cc:504] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.069830 10837 raft_consensus.cc:686] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 LEADER]: Becoming Leader. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Running, Role: LEADER
I20260808 20:12:25.071154 10826 ts_tablet_manager.cc:1428] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260808 20:12:25.072058 10826 tablet_bootstrap.cc:492] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe: Bootstrap starting.
I20260808 20:12:25.073021 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c98a0a1ac84a6aa4b7431b491e4e52" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
I20260808 20:12:25.073567 10644 raft_consensus.cc:3037] T 00c98a0a1ac84a6aa4b7431b491e4e52 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:25.077770 10826 tablet_bootstrap.cc:654] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:25.079663 10644 raft_consensus.cc:2454] T 00c98a0a1ac84a6aa4b7431b491e4e52 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7858bb58840c4165b68c2f114ae20263 in term 1.
I20260808 20:12:25.080798 10481 leader_election.cc:302] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7858bb58840c4165b68c2f114ae20263, ff9f26e2cbef4d1fbf9ca975e1959190; no voters:
I20260808 20:12:25.081633 10884 raft_consensus.cc:2781] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:12:25.082072 10884 raft_consensus.cc:686] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [term 1 LEADER]: Becoming Leader. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Running, Role: LEADER
I20260808 20:12:25.082419 10828 ts_tablet_manager.cc:1428] T b99f754163dc41dbb325bba1ba45c63e P 7858bb58840c4165b68c2f114ae20263: Time spent starting tablet: real 0.019s user 0.005s sys 0.001s
I20260808 20:12:25.083276 10828 tablet_bootstrap.cc:492] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263: Bootstrap starting.
I20260808 20:12:25.082826 10884 consensus_queue.cc:237] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.071138 10837 consensus_queue.cc:237] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.090631 10855 leader_election.cc:288] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 election: Requested vote from peers ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:25.092617 10828 tablet_bootstrap.cc:654] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:25.097681 10760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c98a0a1ac84a6aa4b7431b491e4e52" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186c827b8b48447b94beb413077d76fe"
W20260808 20:12:25.102828 10480 leader_election.cc:341] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:12:25.113602 10826 tablet_bootstrap.cc:492] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe: No bootstrap required, opened a new log
I20260808 20:12:25.114020 10826 ts_tablet_manager.cc:1397] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.027s
I20260808 20:12:25.116772 10827 tablet_bootstrap.cc:492] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190: No bootstrap required, opened a new log
I20260808 20:12:25.117159 10827 ts_tablet_manager.cc:1397] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190: Time spent bootstrapping tablet: real 0.059s user 0.015s sys 0.009s
I20260808 20:12:25.120172 10827 raft_consensus.cc:348] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.120885 10827 raft_consensus.cc:374] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:25.121218 10827 raft_consensus.cc:729] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Initialized, Role: FOLLOWER
I20260808 20:12:25.120750 10389 catalog_manager.cc:5495] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:25.121517 10826 raft_consensus.cc:348] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.121831 10827 consensus_queue.cc:260] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.123466 10826 raft_consensus.cc:374] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:25.125590 10826 raft_consensus.cc:729] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186c827b8b48447b94beb413077d76fe, State: Initialized, Role: FOLLOWER
I20260808 20:12:25.126204 10826 consensus_queue.cc:260] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.126983 10827 ts_tablet_manager.cc:1428] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260808 20:12:25.123541 10384 catalog_manager.cc:5495] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7858bb58840c4165b68c2f114ae20263 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "7858bb58840c4165b68c2f114ae20263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:25.127935 10827 tablet_bootstrap.cc:492] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190: Bootstrap starting.
I20260808 20:12:25.128779 10857 raft_consensus.cc:482] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:25.129254 10857 raft_consensus.cc:504] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.131430 10857 leader_election.cc:288] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407)
I20260808 20:12:25.133551 10826 ts_tablet_manager.cc:1428] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe: Time spent starting tablet: real 0.019s user 0.003s sys 0.003s
I20260808 20:12:25.133440 10532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afafd38c69d4fa5b180f254a38ee912" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7858bb58840c4165b68c2f114ae20263" is_pre_election: true
I20260808 20:12:25.133739 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afafd38c69d4fa5b180f254a38ee912" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" is_pre_election: true
I20260808 20:12:25.134316 10644 raft_consensus.cc:2454] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186c827b8b48447b94beb413077d76fe in term 0.
I20260808 20:12:25.135928 10828 tablet_bootstrap.cc:492] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263: No bootstrap required, opened a new log
I20260808 20:12:25.136365 10828 ts_tablet_manager.cc:1397] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263: Time spent bootstrapping tablet: real 0.053s user 0.014s sys 0.000s
W20260808 20:12:25.135650 10705 leader_election.cc:341] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:12:25.137065 10704 leader_election.cc:302] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, ff9f26e2cbef4d1fbf9ca975e1959190; no voters: 7858bb58840c4165b68c2f114ae20263
I20260808 20:12:25.138375 10833 consensus_queue.cc:1035] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:25.138782 10857 raft_consensus.cc:2781] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:12:25.139205 10857 raft_consensus.cc:482] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:12:25.139593 10857 raft_consensus.cc:3037] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:25.140565 10828 raft_consensus.cc:348] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.141222 10828 raft_consensus.cc:374] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:25.141484 10828 raft_consensus.cc:729] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Initialized, Role: FOLLOWER
I20260808 20:12:25.142050 10828 consensus_queue.cc:260] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.145097 10827 tablet_bootstrap.cc:654] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:25.154160 10857 raft_consensus.cc:504] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.156069 10857 leader_election.cc:288] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 1 election: Requested vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407)
I20260808 20:12:25.156839 10532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afafd38c69d4fa5b180f254a38ee912" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7858bb58840c4165b68c2f114ae20263"
I20260808 20:12:25.157101 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afafd38c69d4fa5b180f254a38ee912" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
I20260808 20:12:25.157985 10644 raft_consensus.cc:3037] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:25.157433 10532 raft_consensus.cc:3037] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:25.169492 10828 ts_tablet_manager.cc:1428] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263: Time spent starting tablet: real 0.033s user 0.002s sys 0.008s
I20260808 20:12:25.172784 10532 raft_consensus.cc:2454] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186c827b8b48447b94beb413077d76fe in term 1.
I20260808 20:12:25.175446 10705 leader_election.cc:302] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, 7858bb58840c4165b68c2f114ae20263; no voters:
I20260808 20:12:25.176133 10857 raft_consensus.cc:2781] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:12:25.171753 10644 raft_consensus.cc:2454] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186c827b8b48447b94beb413077d76fe in term 1.
I20260808 20:12:25.180621 10857 raft_consensus.cc:686] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 1 LEADER]: Becoming Leader. State: Replica: 186c827b8b48447b94beb413077d76fe, State: Running, Role: LEADER
I20260808 20:12:25.181722 10857 consensus_queue.cc:237] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.184406 10827 tablet_bootstrap.cc:492] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190: No bootstrap required, opened a new log
I20260808 20:12:25.184820 10827 ts_tablet_manager.cc:1397] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190: Time spent bootstrapping tablet: real 0.057s user 0.007s sys 0.021s
I20260808 20:12:25.187093 10827 raft_consensus.cc:348] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.187750 10827 raft_consensus.cc:374] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:25.188022 10827 raft_consensus.cc:729] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Initialized, Role: FOLLOWER
I20260808 20:12:25.188575 10827 consensus_queue.cc:260] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.191023 10827 ts_tablet_manager.cc:1428] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260808 20:12:25.197623 10837 raft_consensus.cc:482] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:25.197556 10833 consensus_queue.cc:1035] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:25.198140 10837 raft_consensus.cc:504] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.200140 10837 leader_election.cc:288] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:25.200770 10529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99f754163dc41dbb325bba1ba45c63e" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7858bb58840c4165b68c2f114ae20263" is_pre_election: true
I20260808 20:12:25.201059 10760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99f754163dc41dbb325bba1ba45c63e" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186c827b8b48447b94beb413077d76fe" is_pre_election: true
I20260808 20:12:25.201612 10760 raft_consensus.cc:2454] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff9f26e2cbef4d1fbf9ca975e1959190 in term 0.
I20260808 20:12:25.202672 10586 leader_election.cc:302] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, ff9f26e2cbef4d1fbf9ca975e1959190; no voters:
I20260808 20:12:25.204622 10529 raft_consensus.cc:2454] T b99f754163dc41dbb325bba1ba45c63e P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff9f26e2cbef4d1fbf9ca975e1959190 in term 0.
I20260808 20:12:25.204988 10385 catalog_manager.cc:5495] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 186c827b8b48447b94beb413077d76fe (127.10.24.3). New cstate: current_term: 1 leader_uuid: "186c827b8b48447b94beb413077d76fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: HEALTHY } } }
I20260808 20:12:25.209836 10837 raft_consensus.cc:2781] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:12:25.210191 10837 raft_consensus.cc:482] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:12:25.210505 10837 raft_consensus.cc:3037] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:25.218917 10837 raft_consensus.cc:504] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.221887 10529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99f754163dc41dbb325bba1ba45c63e" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7858bb58840c4165b68c2f114ae20263"
I20260808 20:12:25.222651 10529 raft_consensus.cc:3037] T b99f754163dc41dbb325bba1ba45c63e P 7858bb58840c4165b68c2f114ae20263 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:25.226402 10760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99f754163dc41dbb325bba1ba45c63e" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186c827b8b48447b94beb413077d76fe"
I20260808 20:12:25.226991 10760 raft_consensus.cc:3037] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:25.229344 10529 raft_consensus.cc:2454] T b99f754163dc41dbb325bba1ba45c63e P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff9f26e2cbef4d1fbf9ca975e1959190 in term 1.
I20260808 20:12:25.230648 10589 leader_election.cc:302] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7858bb58840c4165b68c2f114ae20263, ff9f26e2cbef4d1fbf9ca975e1959190; no voters:
I20260808 20:12:25.231504 10890 raft_consensus.cc:2781] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:12:25.232620 10890 raft_consensus.cc:686] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 LEADER]: Becoming Leader. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Running, Role: LEADER
I20260808 20:12:25.232780 10837 leader_election.cc:288] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 1 election: Requested vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:25.233495 10890 consensus_queue.cc:237] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.234616 10760 raft_consensus.cc:2454] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff9f26e2cbef4d1fbf9ca975e1959190 in term 1.
I20260808 20:12:25.245894 10384 catalog_manager.cc:5495] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:25.252638 10884 consensus_queue.cc:1035] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:25.263180 10884 consensus_queue.cc:1035] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:25.305773 10833 consensus_queue.cc:1035] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:25.311570 10833 consensus_queue.cc:1035] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:25.332252 10890 raft_consensus.cc:482] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:25.332772 10890 raft_consensus.cc:504] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.335078 10890 leader_election.cc:288] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:25.336380 10760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8755efc8014e31a8f711fd137edef0" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186c827b8b48447b94beb413077d76fe" is_pre_election: true
I20260808 20:12:25.337178 10760 raft_consensus.cc:2379] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff9f26e2cbef4d1fbf9ca975e1959190 in current term 1: Already voted for candidate 7858bb58840c4165b68c2f114ae20263 in this term.
I20260808 20:12:25.338923 10529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8755efc8014e31a8f711fd137edef0" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7858bb58840c4165b68c2f114ae20263" is_pre_election: true
I20260808 20:12:25.341315 10589 leader_election.cc:302] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff9f26e2cbef4d1fbf9ca975e1959190; no voters: 186c827b8b48447b94beb413077d76fe, 7858bb58840c4165b68c2f114ae20263
I20260808 20:12:25.342150 10890 raft_consensus.cc:3037] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:25.348627 10890 raft_consensus.cc:2726] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:12:25.356563 10760 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2a552acaffcd45268f344a1aa35cb132"
dest_uuid: "186c827b8b48447b94beb413077d76fe"
mode: GRACEFUL
new_leader_uuid: "7858bb58840c4165b68c2f114ae20263"
from {username='slave'} at 127.0.0.1:58164
I20260808 20:12:25.357105 10760 raft_consensus.cc:595] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [term 1 LEADER]: Received request to transfer leadership to TS 7858bb58840c4165b68c2f114ae20263
I20260808 20:12:25.361804 10760 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10c2df6974bc48a19899f8ef6a1932dc"
dest_uuid: "186c827b8b48447b94beb413077d76fe"
mode: GRACEFUL
new_leader_uuid: "7858bb58840c4165b68c2f114ae20263"
from {username='slave'} at 127.0.0.1:58164
I20260808 20:12:25.362272 10760 raft_consensus.cc:595] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 1 LEADER]: Received request to transfer leadership to TS 7858bb58840c4165b68c2f114ae20263
I20260808 20:12:25.385679 10529 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4c8755efc8014e31a8f711fd137edef0"
dest_uuid: "7858bb58840c4165b68c2f114ae20263"
mode: GRACEFUL
new_leader_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
from {username='slave'} at 127.0.0.1:52536
I20260808 20:12:25.386231 10529 raft_consensus.cc:595] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [term 1 LEADER]: Received request to transfer leadership to TS ff9f26e2cbef4d1fbf9ca975e1959190
I20260808 20:12:25.390498 10760 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3afafd38c69d4fa5b180f254a38ee912"
dest_uuid: "186c827b8b48447b94beb413077d76fe"
mode: GRACEFUL
new_leader_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
from {username='slave'} at 127.0.0.1:58164
I20260808 20:12:25.390954 10760 raft_consensus.cc:595] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 1 LEADER]: Received request to transfer leadership to TS ff9f26e2cbef4d1fbf9ca975e1959190
I20260808 20:12:25.397974 10884 consensus_queue.cc:1035] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:25.398531 10529 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "00c98a0a1ac84a6aa4b7431b491e4e52"
dest_uuid: "7858bb58840c4165b68c2f114ae20263"
mode: GRACEFUL
new_leader_uuid: "186c827b8b48447b94beb413077d76fe"
from {username='slave'} at 127.0.0.1:52536
I20260808 20:12:25.399098 10529 raft_consensus.cc:595] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [term 1 LEADER]: Received request to transfer leadership to TS 186c827b8b48447b94beb413077d76fe
I20260808 20:12:25.407855 10883 consensus_queue.cc:1035] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:12:25.451465 10857 raft_consensus.cc:482] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:25.451982 10857 raft_consensus.cc:504] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.454113 10857 leader_election.cc:288] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407)
I20260808 20:12:25.454941 10529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c98a0a1ac84a6aa4b7431b491e4e52" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7858bb58840c4165b68c2f114ae20263" is_pre_election: true
I20260808 20:12:25.455160 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c98a0a1ac84a6aa4b7431b491e4e52" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" is_pre_election: true
I20260808 20:12:25.455955 10644 raft_consensus.cc:2379] T 00c98a0a1ac84a6aa4b7431b491e4e52 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186c827b8b48447b94beb413077d76fe in current term 1: Already voted for candidate 7858bb58840c4165b68c2f114ae20263 in this term.
I20260808 20:12:25.456879 10704 leader_election.cc:302] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186c827b8b48447b94beb413077d76fe; no voters: 7858bb58840c4165b68c2f114ae20263, ff9f26e2cbef4d1fbf9ca975e1959190
I20260808 20:12:25.457726 10857 raft_consensus.cc:3037] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:25.462059 10857 raft_consensus.cc:2726] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:12:25.470146 10883 consensus_queue.cc:1035] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:25.481308 10855 consensus_queue.cc:1035] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:25.489145 10890 consensus_queue.cc:1035] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:25.493546 10855 consensus_queue.cc:1035] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:25.494344 10835 raft_consensus.cc:979] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263: : Instructing follower ff9f26e2cbef4d1fbf9ca975e1959190 to start an election
I20260808 20:12:25.494870 10855 raft_consensus.cc:1067] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [term 1 LEADER]: Signalling peer ff9f26e2cbef4d1fbf9ca975e1959190 to start an election
I20260808 20:12:25.498873 10644 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4c8755efc8014e31a8f711fd137edef0"
dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
from {username='slave'} at 127.0.0.1:38440
I20260808 20:12:25.499599 10644 raft_consensus.cc:482] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:25.500008 10644 raft_consensus.cc:3037] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:25.508731 10644 raft_consensus.cc:504] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.513765 10883 raft_consensus.cc:979] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263: : Instructing follower 186c827b8b48447b94beb413077d76fe to start an election
I20260808 20:12:25.514124 10644 leader_election.cc:288] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 2 election: Requested vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:25.514269 10883 raft_consensus.cc:1067] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [term 1 LEADER]: Signalling peer 186c827b8b48447b94beb413077d76fe to start an election
I20260808 20:12:25.516389 10529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8755efc8014e31a8f711fd137edef0" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7858bb58840c4165b68c2f114ae20263"
I20260808 20:12:25.519243 10529 raft_consensus.cc:3032] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:12:25.519565 10529 raft_consensus.cc:729] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Running, Role: LEADER
I20260808 20:12:25.519734 10760 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00c98a0a1ac84a6aa4b7431b491e4e52"
dest_uuid: "186c827b8b48447b94beb413077d76fe"
from {username='slave'} at 127.0.0.1:58132
I20260808 20:12:25.520361 10760 raft_consensus.cc:482] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:25.520820 10760 raft_consensus.cc:3037] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:25.520298 10529 consensus_queue.cc:260] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.527599 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8755efc8014e31a8f711fd137edef0" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "186c827b8b48447b94beb413077d76fe"
I20260808 20:12:25.527921 10837 consensus_queue.cc:1035] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:25.530752 10835 consensus_queue.cc:1035] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:25.528195 10761 raft_consensus.cc:3037] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:25.527750 10529 raft_consensus.cc:3037] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:25.541287 10761 raft_consensus.cc:2454] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff9f26e2cbef4d1fbf9ca975e1959190 in term 2.
I20260808 20:12:25.541502 10529 raft_consensus.cc:2454] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff9f26e2cbef4d1fbf9ca975e1959190 in term 2.
I20260808 20:12:25.542582 10589 leader_election.cc:302] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7858bb58840c4165b68c2f114ae20263, ff9f26e2cbef4d1fbf9ca975e1959190; no voters:
I20260808 20:12:25.543408 10837 raft_consensus.cc:2781] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:12:25.542606 10760 raft_consensus.cc:504] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.544023 10837 raft_consensus.cc:686] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 LEADER]: Becoming Leader. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Running, Role: LEADER
I20260808 20:12:25.545009 10837 consensus_queue.cc:237] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.552698 10529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c98a0a1ac84a6aa4b7431b491e4e52" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7858bb58840c4165b68c2f114ae20263"
I20260808 20:12:25.553359 10529 raft_consensus.cc:3032] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:12:25.555197 10645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c98a0a1ac84a6aa4b7431b491e4e52" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
I20260808 20:12:25.557489 10529 raft_consensus.cc:729] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Running, Role: LEADER
I20260808 20:12:25.557785 10760 leader_election.cc:288] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 2 election: Requested vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407)
I20260808 20:12:25.559751 10529 consensus_queue.cc:260] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.560963 10529 raft_consensus.cc:3037] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:25.567062 10529 raft_consensus.cc:2454] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186c827b8b48447b94beb413077d76fe in term 2.
I20260808 20:12:25.567844 10705 leader_election.cc:302] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, 7858bb58840c4165b68c2f114ae20263; no voters: ff9f26e2cbef4d1fbf9ca975e1959190
I20260808 20:12:25.568960 10857 raft_consensus.cc:2781] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:12:25.569435 10857 raft_consensus.cc:686] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe [term 2 LEADER]: Becoming Leader. State: Replica: 186c827b8b48447b94beb413077d76fe, State: Running, Role: LEADER
I20260808 20:12:25.570233 10857 consensus_queue.cc:237] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.578243 10389 catalog_manager.cc:5495] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 reported cstate change: term changed from 1 to 2, leader changed from 7858bb58840c4165b68c2f114ae20263 (127.10.24.1) to ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2). New cstate: current_term: 2 leader_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:25.587139 10385 catalog_manager.cc:5495] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe reported cstate change: term changed from 1 to 2, leader changed from 7858bb58840c4165b68c2f114ae20263 (127.10.24.1) to 186c827b8b48447b94beb413077d76fe (127.10.24.3). New cstate: current_term: 2 leader_uuid: "186c827b8b48447b94beb413077d76fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: HEALTHY } } }
I20260808 20:12:25.753739 10837 consensus_queue.cc:1035] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:25.763185 10890 consensus_queue.cc:1035] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:25.785844 10909 raft_consensus.cc:979] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe: : Instructing follower 7858bb58840c4165b68c2f114ae20263 to start an election
I20260808 20:12:25.786350 10857 raft_consensus.cc:1067] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [term 1 LEADER]: Signalling peer 7858bb58840c4165b68c2f114ae20263 to start an election
I20260808 20:12:25.787837 10529 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2a552acaffcd45268f344a1aa35cb132"
dest_uuid: "7858bb58840c4165b68c2f114ae20263"
from {username='slave'} at 127.0.0.1:52516
I20260808 20:12:25.788352 10529 raft_consensus.cc:482] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:25.788715 10529 raft_consensus.cc:3037] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:25.795189 10529 raft_consensus.cc:504] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } } peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } }
I20260808 20:12:25.798463 10529 leader_election.cc:288] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 2 election: Requested vote from peers 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651), ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407)
I20260808 20:12:25.799014 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a552acaffcd45268f344a1aa35cb132" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
I20260808 20:12:25.799556 10644 raft_consensus.cc:3037] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:25.799791 10760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a552acaffcd45268f344a1aa35cb132" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "186c827b8b48447b94beb413077d76fe"
I20260808 20:12:25.800758 10760 raft_consensus.cc:3032] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:12:25.801077 10760 raft_consensus.cc:729] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 186c827b8b48447b94beb413077d76fe, State: Running, Role: LEADER
I20260808 20:12:25.801887 10760 consensus_queue.cc:260] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } } peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } }
I20260808 20:12:25.803210 10760 raft_consensus.cc:3037] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:25.807418 10644 raft_consensus.cc:2454] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7858bb58840c4165b68c2f114ae20263 in term 2.
I20260808 20:12:25.810536 10481 leader_election.cc:302] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7858bb58840c4165b68c2f114ae20263, ff9f26e2cbef4d1fbf9ca975e1959190; no voters:
I20260808 20:12:25.811097 10760 raft_consensus.cc:2454] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7858bb58840c4165b68c2f114ae20263 in term 2.
I20260808 20:12:25.811305 10835 raft_consensus.cc:2781] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:12:25.811793 10835 raft_consensus.cc:686] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [term 2 LEADER]: Becoming Leader. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Running, Role: LEADER
I20260808 20:12:25.812004 10857 consensus_queue.cc:1035] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:25.812561 10835 consensus_queue.cc:237] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } } peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } }
I20260808 20:12:25.818244 10857 consensus_queue.cc:1035] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:25.824229 10857 raft_consensus.cc:979] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe: : Instructing follower ff9f26e2cbef4d1fbf9ca975e1959190 to start an election
I20260808 20:12:25.824632 10857 raft_consensus.cc:1067] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 1 LEADER]: Signalling peer ff9f26e2cbef4d1fbf9ca975e1959190 to start an election
I20260808 20:12:25.832875 10639 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3afafd38c69d4fa5b180f254a38ee912"
dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
from {username='slave'} at 127.0.0.1:38446
I20260808 20:12:25.834180 10639 raft_consensus.cc:482] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:25.834581 10639 raft_consensus.cc:3037] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:25.836301 10389 catalog_manager.cc:5495] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 reported cstate change: term changed from 1 to 2, leader changed from 186c827b8b48447b94beb413077d76fe (127.10.24.3) to 7858bb58840c4165b68c2f114ae20263 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "7858bb58840c4165b68c2f114ae20263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:25.842674 10639 raft_consensus.cc:504] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.845302 10532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afafd38c69d4fa5b180f254a38ee912" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7858bb58840c4165b68c2f114ae20263"
I20260808 20:12:25.845826 10532 raft_consensus.cc:3037] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:25.845886 10639 leader_election.cc:288] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 2 election: Requested vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:25.848484 10760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afafd38c69d4fa5b180f254a38ee912" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "186c827b8b48447b94beb413077d76fe"
I20260808 20:12:25.851923 10532 raft_consensus.cc:2454] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff9f26e2cbef4d1fbf9ca975e1959190 in term 2.
I20260808 20:12:25.853071 10589 leader_election.cc:302] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7858bb58840c4165b68c2f114ae20263, ff9f26e2cbef4d1fbf9ca975e1959190; no voters:
I20260808 20:12:25.853763 10890 raft_consensus.cc:2781] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:12:25.854272 10890 raft_consensus.cc:686] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 LEADER]: Becoming Leader. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Running, Role: LEADER
I20260808 20:12:25.855074 10890 consensus_queue.cc:237] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.851256 10760 raft_consensus.cc:3032] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:12:25.857113 10760 raft_consensus.cc:729] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 186c827b8b48447b94beb413077d76fe, State: Running, Role: LEADER
I20260808 20:12:25.857960 10760 consensus_queue.cc:260] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.859261 10760 raft_consensus.cc:3037] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:25.863843 10384 catalog_manager.cc:5495] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 reported cstate change: term changed from 1 to 2, leader changed from 186c827b8b48447b94beb413077d76fe (127.10.24.3) to ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2). New cstate: current_term: 2 leader_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:25.865147 10760 raft_consensus.cc:2454] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff9f26e2cbef4d1fbf9ca975e1959190 in term 2.
I20260808 20:12:25.924783 10833 raft_consensus.cc:979] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe: : Instructing follower 7858bb58840c4165b68c2f114ae20263 to start an election
I20260808 20:12:25.925253 10857 raft_consensus.cc:1067] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 1 LEADER]: Signalling peer 7858bb58840c4165b68c2f114ae20263 to start an election
I20260808 20:12:25.926561 10532 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10c2df6974bc48a19899f8ef6a1932dc"
dest_uuid: "7858bb58840c4165b68c2f114ae20263"
from {username='slave'} at 127.0.0.1:52516
I20260808 20:12:25.927047 10532 raft_consensus.cc:482] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:25.927361 10532 raft_consensus.cc:3037] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:25.931913 10532 raft_consensus.cc:504] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.933748 10532 leader_election.cc:288] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 2 election: Requested vote from peers ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:25.934907 10760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c2df6974bc48a19899f8ef6a1932dc" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "186c827b8b48447b94beb413077d76fe"
I20260808 20:12:25.935232 10639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c2df6974bc48a19899f8ef6a1932dc" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
I20260808 20:12:25.935464 10760 raft_consensus.cc:3032] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:12:25.935812 10760 raft_consensus.cc:729] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 186c827b8b48447b94beb413077d76fe, State: Running, Role: LEADER
I20260808 20:12:25.936457 10760 consensus_queue.cc:260] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.937727 10760 raft_consensus.cc:3037] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:25.941720 10760 raft_consensus.cc:2454] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7858bb58840c4165b68c2f114ae20263 in term 2.
I20260808 20:12:25.942548 10480 leader_election.cc:302] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, 7858bb58840c4165b68c2f114ae20263; no voters: ff9f26e2cbef4d1fbf9ca975e1959190
I20260808 20:12:25.943140 10884 raft_consensus.cc:2781] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:12:25.943540 10884 raft_consensus.cc:686] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 2 LEADER]: Becoming Leader. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Running, Role: LEADER
I20260808 20:12:25.944188 10884 consensus_queue.cc:237] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:25.950057 10384 catalog_manager.cc:5495] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 reported cstate change: term changed from 1 to 2, leader changed from 186c827b8b48447b94beb413077d76fe (127.10.24.3) to 7858bb58840c4165b68c2f114ae20263 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "7858bb58840c4165b68c2f114ae20263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:25.955960 10532 raft_consensus.cc:1261] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Refusing update from remote peer 186c827b8b48447b94beb413077d76fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:25.958251 10857 consensus_queue.cc:1035] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:25.965349 10639 raft_consensus.cc:3037] T 00c98a0a1ac84a6aa4b7431b491e4e52 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:25.973256 10639 raft_consensus.cc:1261] T 00c98a0a1ac84a6aa4b7431b491e4e52 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Refusing update from remote peer 186c827b8b48447b94beb413077d76fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:25.974872 10909 consensus_queue.cc:1035] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:25.988300 10760 raft_consensus.cc:1261] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Refusing update from remote peer ff9f26e2cbef4d1fbf9ca975e1959190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:25.989356 10890 consensus_queue.cc:1035] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:25.996158 10532 raft_consensus.cc:1261] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Refusing update from remote peer ff9f26e2cbef4d1fbf9ca975e1959190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:25.998620 10914 consensus_queue.cc:1035] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:26.280208 10639 raft_consensus.cc:1261] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Refusing update from remote peer 7858bb58840c4165b68c2f114ae20263: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:26.281335 10884 consensus_queue.cc:1035] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:26.288498 10532 raft_consensus.cc:1261] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Refusing update from remote peer ff9f26e2cbef4d1fbf9ca975e1959190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:26.289095 10760 raft_consensus.cc:1261] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Refusing update from remote peer 7858bb58840c4165b68c2f114ae20263: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:26.289885 10914 consensus_queue.cc:1035] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:26.291846 10835 consensus_queue.cc:1035] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:26.298020 10761 raft_consensus.cc:1261] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Refusing update from remote peer ff9f26e2cbef4d1fbf9ca975e1959190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:26.299446 10914 consensus_queue.cc:1035] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:26.553030 10760 raft_consensus.cc:1261] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Refusing update from remote peer 7858bb58840c4165b68c2f114ae20263: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:26.554140 10855 consensus_queue.cc:1035] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:26.561560 10639 raft_consensus.cc:3037] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:26.567850 10639 raft_consensus.cc:1261] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Refusing update from remote peer 7858bb58840c4165b68c2f114ae20263: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:26.569042 10884 consensus_queue.cc:1035] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:28.425978 10336 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20260808 20:12:28.426733 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:12:28.446141 10645 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3afafd38c69d4fa5b180f254a38ee912"
dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
mode: GRACEFUL
new_leader_uuid: "186c827b8b48447b94beb413077d76fe"
from {username='slave'} at 127.0.0.1:38460
I20260808 20:12:28.446602 10645 raft_consensus.cc:595] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 LEADER]: Received request to transfer leadership to TS 186c827b8b48447b94beb413077d76fe
I20260808 20:12:28.457221 10532 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b193cc24b354319ab91439036f51881"
dest_uuid: "7858bb58840c4165b68c2f114ae20263"
mode: GRACEFUL
new_leader_uuid: "186c827b8b48447b94beb413077d76fe"
from {username='slave'} at 127.0.0.1:52542
I20260808 20:12:28.457675 10532 raft_consensus.cc:595] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [term 1 LEADER]: Received request to transfer leadership to TS 186c827b8b48447b94beb413077d76fe
I20260808 20:12:28.458600 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260808 20:12:28.458990 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:12:28.474529 10959 raft_consensus.cc:979] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190: : Instructing follower 186c827b8b48447b94beb413077d76fe to start an election
I20260808 20:12:28.475023 10954 raft_consensus.cc:1067] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 LEADER]: Signalling peer 186c827b8b48447b94beb413077d76fe to start an election
I20260808 20:12:28.476310 10760 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3afafd38c69d4fa5b180f254a38ee912"
dest_uuid: "186c827b8b48447b94beb413077d76fe"
from {username='slave'} at 127.0.0.1:58148
I20260808 20:12:28.476738 10760 raft_consensus.cc:482] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:28.477039 10760 raft_consensus.cc:3037] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:12:28.481798 10760 raft_consensus.cc:504] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:28.483209 10760 leader_election.cc:288] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 3 election: Requested vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407)
I20260808 20:12:28.483794 10532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afafd38c69d4fa5b180f254a38ee912" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7858bb58840c4165b68c2f114ae20263"
I20260808 20:12:28.484359 10532 raft_consensus.cc:3037] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:12:28.484274 10645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afafd38c69d4fa5b180f254a38ee912" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
I20260808 20:12:28.484879 10645 raft_consensus.cc:3032] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:12:28.485214 10645 raft_consensus.cc:729] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Running, Role: LEADER
I20260808 20:12:28.485950 10645 consensus_queue.cc:260] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:28.487112 10645 raft_consensus.cc:3037] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:12:28.490314 10532 raft_consensus.cc:2454] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186c827b8b48447b94beb413077d76fe in term 3.
I20260808 20:12:28.491364 10705 leader_election.cc:302] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, 7858bb58840c4165b68c2f114ae20263; no voters:
I20260808 20:12:28.492094 10952 raft_consensus.cc:2781] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:12:28.492663 10645 raft_consensus.cc:2454] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186c827b8b48447b94beb413077d76fe in term 3.
I20260808 20:12:28.493273 10952 raft_consensus.cc:686] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 3 LEADER]: Becoming Leader. State: Replica: 186c827b8b48447b94beb413077d76fe, State: Running, Role: LEADER
I20260808 20:12:28.494117 10952 consensus_queue.cc:237] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:28.500052 10384 catalog_manager.cc:5495] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe reported cstate change: term changed from 2 to 3, leader changed from ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2) to 186c827b8b48447b94beb413077d76fe (127.10.24.3). New cstate: current_term: 3 leader_uuid: "186c827b8b48447b94beb413077d76fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: HEALTHY } } }
I20260808 20:12:28.935096 10532 raft_consensus.cc:1261] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [term 3 FOLLOWER]: Refusing update from remote peer 186c827b8b48447b94beb413077d76fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:12:28.936223 10952 consensus_queue.cc:1035] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:12:28.936347 10645 raft_consensus.cc:1261] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 FOLLOWER]: Refusing update from remote peer 186c827b8b48447b94beb413077d76fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:12:28.937548 10952 consensus_queue.cc:1035] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:12:28.943948 10974 raft_consensus.cc:979] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263: : Instructing follower 186c827b8b48447b94beb413077d76fe to start an election
I20260808 20:12:28.944339 10974 raft_consensus.cc:1067] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [term 1 LEADER]: Signalling peer 186c827b8b48447b94beb413077d76fe to start an election
I20260808 20:12:28.946043 10760 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b193cc24b354319ab91439036f51881"
dest_uuid: "186c827b8b48447b94beb413077d76fe"
from {username='slave'} at 127.0.0.1:58132
I20260808 20:12:28.946565 10760 raft_consensus.cc:482] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:28.946867 10760 raft_consensus.cc:3037] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:28.952461 10760 raft_consensus.cc:504] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:28.953794 10760 leader_election.cc:288] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 2 election: Requested vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407)
I20260808 20:12:28.955523 10639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b193cc24b354319ab91439036f51881" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
I20260808 20:12:28.956085 10639 raft_consensus.cc:3037] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:28.956879 10529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b193cc24b354319ab91439036f51881" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7858bb58840c4165b68c2f114ae20263"
I20260808 20:12:28.957518 10529 raft_consensus.cc:3032] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:12:28.957808 10529 raft_consensus.cc:729] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Running, Role: LEADER
I20260808 20:12:28.958484 10529 consensus_queue.cc:260] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:28.959543 10529 raft_consensus.cc:3037] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:28.963618 10639 raft_consensus.cc:2454] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186c827b8b48447b94beb413077d76fe in term 2.
I20260808 20:12:28.964589 10704 leader_election.cc:302] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, ff9f26e2cbef4d1fbf9ca975e1959190; no voters:
I20260808 20:12:28.965451 10980 raft_consensus.cc:2781] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:12:28.965891 10980 raft_consensus.cc:686] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [term 2 LEADER]: Becoming Leader. State: Replica: 186c827b8b48447b94beb413077d76fe, State: Running, Role: LEADER
I20260808 20:12:28.965921 10529 raft_consensus.cc:2454] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186c827b8b48447b94beb413077d76fe in term 2.
I20260808 20:12:28.966809 10980 consensus_queue.cc:237] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:28.973901 10385 catalog_manager.cc:5495] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe reported cstate change: term changed from 1 to 2, leader changed from 7858bb58840c4165b68c2f114ae20263 (127.10.24.1) to 186c827b8b48447b94beb413077d76fe (127.10.24.3). New cstate: current_term: 2 leader_uuid: "186c827b8b48447b94beb413077d76fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: HEALTHY } } }
I20260808 20:12:29.417661 10639 raft_consensus.cc:1261] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Refusing update from remote peer 186c827b8b48447b94beb413077d76fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:29.418905 10980 consensus_queue.cc:1035] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:29.430500 10529 raft_consensus.cc:1261] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Refusing update from remote peer 186c827b8b48447b94beb413077d76fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:29.431712 10987 consensus_queue.cc:1035] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:29.459712 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:12:29.464398 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:12:29.464730 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:12:30.465415 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:12:30.470009 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:12:30.470355 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:12:31.471060 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:12:31.476205 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:12:31.476568 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:12:32.477430 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:12:32.482031 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:12:32.482434 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:12:33.483184 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:12:33.489310 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:12:33.489751 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:12:34.490449 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:12:34.496018 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:12:34.496418 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:12:35.497251 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:12:35.501734 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:12:35.502247 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:12:36.503024 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:12:36.507835 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:12:36.508157 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:12:37.508863 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:12:37.513640 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:12:37.513973 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:12:38.536640 10336 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20260808 20:12:38.540688 10529 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2a552acaffcd45268f344a1aa35cb132"
dest_uuid: "7858bb58840c4165b68c2f114ae20263"
mode: GRACEFUL
new_leader_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
from {username='slave'} at 127.0.0.1:52536
I20260808 20:12:38.541430 10529 raft_consensus.cc:595] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [term 2 LEADER]: Received request to transfer leadership to TS ff9f26e2cbef4d1fbf9ca975e1959190
I20260808 20:12:39.213169 10999 raft_consensus.cc:979] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263: : Instructing follower ff9f26e2cbef4d1fbf9ca975e1959190 to start an election
I20260808 20:12:39.213920 10999 raft_consensus.cc:1067] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [term 2 LEADER]: Signalling peer ff9f26e2cbef4d1fbf9ca975e1959190 to start an election
I20260808 20:12:39.216289 10643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2a552acaffcd45268f344a1aa35cb132"
dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
from {username='slave'} at 127.0.0.1:38440
I20260808 20:12:39.216954 10643 raft_consensus.cc:482] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:39.216995 10532 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10c2df6974bc48a19899f8ef6a1932dc"
dest_uuid: "7858bb58840c4165b68c2f114ae20263"
mode: GRACEFUL
new_leader_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
from {username='slave'} at 127.0.0.1:52536
I20260808 20:12:39.217515 10643 raft_consensus.cc:3037] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:12:39.222011 10532 raft_consensus.cc:595] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 2 LEADER]: Received request to transfer leadership to TS ff9f26e2cbef4d1fbf9ca975e1959190
I20260808 20:12:39.226927 10643 raft_consensus.cc:504] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } } peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } }
I20260808 20:12:39.229961 10643 leader_election.cc:288] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 3 election: Requested vote from peers 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651), 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783)
I20260808 20:12:39.230238 10760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a552acaffcd45268f344a1aa35cb132" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "186c827b8b48447b94beb413077d76fe"
I20260808 20:12:39.230993 10760 raft_consensus.cc:3037] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:12:39.230911 10529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a552acaffcd45268f344a1aa35cb132" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7858bb58840c4165b68c2f114ae20263"
I20260808 20:12:39.231521 10529 raft_consensus.cc:3032] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:12:39.231756 10759 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b193cc24b354319ab91439036f51881"
dest_uuid: "186c827b8b48447b94beb413077d76fe"
mode: GRACEFUL
new_leader_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
from {username='slave'} at 127.0.0.1:58164
I20260808 20:12:39.231956 10529 raft_consensus.cc:729] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Running, Role: LEADER
I20260808 20:12:39.232345 10759 raft_consensus.cc:595] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [term 2 LEADER]: Received request to transfer leadership to TS ff9f26e2cbef4d1fbf9ca975e1959190
I20260808 20:12:39.232673 10529 consensus_queue.cc:260] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } } peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } }
I20260808 20:12:39.233795 10529 raft_consensus.cc:3037] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:12:39.237076 10760 raft_consensus.cc:2454] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff9f26e2cbef4d1fbf9ca975e1959190 in term 3.
I20260808 20:12:39.237663 10532 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "94846b349249412faaa65fb6ef97df95"
dest_uuid: "7858bb58840c4165b68c2f114ae20263"
mode: GRACEFUL
new_leader_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
from {username='slave'} at 127.0.0.1:52536
I20260808 20:12:39.238297 10586 leader_election.cc:302] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, ff9f26e2cbef4d1fbf9ca975e1959190; no voters:
I20260808 20:12:39.238387 10532 raft_consensus.cc:595] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [term 1 LEADER]: Received request to transfer leadership to TS ff9f26e2cbef4d1fbf9ca975e1959190
I20260808 20:12:39.239293 10962 raft_consensus.cc:2781] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:12:39.239673 10529 raft_consensus.cc:2454] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff9f26e2cbef4d1fbf9ca975e1959190 in term 3.
I20260808 20:12:39.239786 10962 raft_consensus.cc:686] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 LEADER]: Becoming Leader. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Running, Role: LEADER
I20260808 20:12:39.241576 10994 raft_consensus.cc:979] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe: : Instructing follower ff9f26e2cbef4d1fbf9ca975e1959190 to start an election
I20260808 20:12:39.242293 10994 raft_consensus.cc:1067] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [term 2 LEADER]: Signalling peer ff9f26e2cbef4d1fbf9ca975e1959190 to start an election
I20260808 20:12:39.242537 10962 consensus_queue.cc:237] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } } peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } }
I20260808 20:12:39.245434 10643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b193cc24b354319ab91439036f51881"
dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
from {username='slave'} at 127.0.0.1:38446
I20260808 20:12:39.246044 10643 raft_consensus.cc:482] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:39.246454 10643 raft_consensus.cc:3037] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:12:39.250098 10760 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3afafd38c69d4fa5b180f254a38ee912"
dest_uuid: "186c827b8b48447b94beb413077d76fe"
mode: GRACEFUL
new_leader_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
from {username='slave'} at 127.0.0.1:58164
I20260808 20:12:39.250509 10760 raft_consensus.cc:595] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 3 LEADER]: Received request to transfer leadership to TS ff9f26e2cbef4d1fbf9ca975e1959190
I20260808 20:12:39.252686 10643 raft_consensus.cc:504] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:39.254588 10643 leader_election.cc:288] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 3 election: Requested vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:39.255344 10389 catalog_manager.cc:5495] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 reported cstate change: term changed from 2 to 3, leader changed from 7858bb58840c4165b68c2f114ae20263 (127.10.24.1) to ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2). New cstate: current_term: 3 leader_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: HEALTHY } } }
I20260808 20:12:39.255939 10760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b193cc24b354319ab91439036f51881" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "186c827b8b48447b94beb413077d76fe"
I20260808 20:12:39.256233 10529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b193cc24b354319ab91439036f51881" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7858bb58840c4165b68c2f114ae20263"
I20260808 20:12:39.256539 10760 raft_consensus.cc:3032] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:12:39.256742 10529 raft_consensus.cc:3037] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:12:39.256878 10760 raft_consensus.cc:729] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 186c827b8b48447b94beb413077d76fe, State: Running, Role: LEADER
I20260808 20:12:39.257562 10760 consensus_queue.cc:260] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:39.258483 10760 raft_consensus.cc:3037] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:12:39.260830 10529 raft_consensus.cc:2454] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff9f26e2cbef4d1fbf9ca975e1959190 in term 3.
I20260808 20:12:39.262261 10760 raft_consensus.cc:2454] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff9f26e2cbef4d1fbf9ca975e1959190 in term 3.
I20260808 20:12:39.262737 10589 leader_election.cc:302] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7858bb58840c4165b68c2f114ae20263, ff9f26e2cbef4d1fbf9ca975e1959190; no voters:
I20260808 20:12:39.263463 10962 raft_consensus.cc:2781] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:12:39.263928 10962 raft_consensus.cc:686] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 LEADER]: Becoming Leader. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Running, Role: LEADER
I20260808 20:12:39.264747 10962 consensus_queue.cc:237] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:39.278177 10389 catalog_manager.cc:5495] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 reported cstate change: term changed from 2 to 3, leader changed from 186c827b8b48447b94beb413077d76fe (127.10.24.3) to ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2). New cstate: current_term: 3 leader_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:39.591140 10999 raft_consensus.cc:979] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263: : Instructing follower ff9f26e2cbef4d1fbf9ca975e1959190 to start an election
I20260808 20:12:39.591611 11000 raft_consensus.cc:1067] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 2 LEADER]: Signalling peer ff9f26e2cbef4d1fbf9ca975e1959190 to start an election
I20260808 20:12:39.592871 10643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10c2df6974bc48a19899f8ef6a1932dc"
dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
from {username='slave'} at 127.0.0.1:38440
I20260808 20:12:39.593382 10643 raft_consensus.cc:482] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:39.593663 10643 raft_consensus.cc:3037] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:12:39.597863 10643 raft_consensus.cc:504] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:39.599231 10643 leader_election.cc:288] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 3 election: Requested vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:39.599968 10529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c2df6974bc48a19899f8ef6a1932dc" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7858bb58840c4165b68c2f114ae20263"
I20260808 20:12:39.600229 10760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c2df6974bc48a19899f8ef6a1932dc" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "186c827b8b48447b94beb413077d76fe"
I20260808 20:12:39.600433 10529 raft_consensus.cc:3032] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:12:39.600668 10760 raft_consensus.cc:3037] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:12:39.600690 10529 raft_consensus.cc:729] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Running, Role: LEADER
I20260808 20:12:39.601454 10529 consensus_queue.cc:260] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:39.602283 10529 raft_consensus.cc:3037] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:12:39.604686 10760 raft_consensus.cc:2454] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff9f26e2cbef4d1fbf9ca975e1959190 in term 3.
I20260808 20:12:39.605456 10586 leader_election.cc:302] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, ff9f26e2cbef4d1fbf9ca975e1959190; no voters:
I20260808 20:12:39.606070 10962 raft_consensus.cc:2781] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:12:39.606330 10529 raft_consensus.cc:2454] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff9f26e2cbef4d1fbf9ca975e1959190 in term 3.
I20260808 20:12:39.606384 10962 raft_consensus.cc:686] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 LEADER]: Becoming Leader. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Running, Role: LEADER
I20260808 20:12:39.607102 10962 consensus_queue.cc:237] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:39.614437 10389 catalog_manager.cc:5495] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 reported cstate change: term changed from 2 to 3, leader changed from 7858bb58840c4165b68c2f114ae20263 (127.10.24.1) to ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2). New cstate: current_term: 3 leader_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:39.626920 10999 raft_consensus.cc:979] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263: : Instructing follower ff9f26e2cbef4d1fbf9ca975e1959190 to start an election
I20260808 20:12:39.627315 11000 raft_consensus.cc:1067] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [term 1 LEADER]: Signalling peer ff9f26e2cbef4d1fbf9ca975e1959190 to start an election
I20260808 20:12:39.628501 11002 raft_consensus.cc:979] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe: : Instructing follower ff9f26e2cbef4d1fbf9ca975e1959190 to start an election
I20260808 20:12:39.628729 10643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "94846b349249412faaa65fb6ef97df95"
dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
from {username='slave'} at 127.0.0.1:38440
I20260808 20:12:39.629066 11001 raft_consensus.cc:1067] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 3 LEADER]: Signalling peer ff9f26e2cbef4d1fbf9ca975e1959190 to start an election
I20260808 20:12:39.629297 10643 raft_consensus.cc:482] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:39.629675 10643 raft_consensus.cc:3037] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:39.630532 10645 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3afafd38c69d4fa5b180f254a38ee912"
dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
from {username='slave'} at 127.0.0.1:38446
I20260808 20:12:39.630926 10645 raft_consensus.cc:482] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:39.631184 10645 raft_consensus.cc:3037] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:12:39.630072 10794 debug-util.cc:398] Leaking SignalData structure 0x7b0800175060 after lost signal to thread 10340
W20260808 20:12:39.632112 10794 debug-util.cc:398] Leaking SignalData structure 0x7b0800175740 after lost signal to thread 10435
W20260808 20:12:39.633087 10794 debug-util.cc:398] Leaking SignalData structure 0x7b0800150a80 after lost signal to thread 10564
W20260808 20:12:39.633942 10794 debug-util.cc:398] Leaking SignalData structure 0x7b08001f9c00 after lost signal to thread 10683
W20260808 20:12:39.634802 10794 debug-util.cc:398] Leaking SignalData structure 0x7b08001df9a0 after lost signal to thread 10798
I20260808 20:12:39.634567 10643 raft_consensus.cc:504] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:39.636576 10643 leader_election.cc:288] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 2 election: Requested vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:39.636844 10529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94846b349249412faaa65fb6ef97df95" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7858bb58840c4165b68c2f114ae20263"
I20260808 20:12:39.637357 10529 raft_consensus.cc:3032] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:12:39.637567 10529 raft_consensus.cc:729] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Running, Role: LEADER
I20260808 20:12:39.637843 10645 raft_consensus.cc:504] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:39.638226 10760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94846b349249412faaa65fb6ef97df95" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "186c827b8b48447b94beb413077d76fe"
I20260808 20:12:39.638128 10529 consensus_queue.cc:260] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:39.638717 10760 raft_consensus.cc:3037] T 94846b349249412faaa65fb6ef97df95 P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:39.639166 10529 raft_consensus.cc:3037] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:39.640473 10532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afafd38c69d4fa5b180f254a38ee912" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "7858bb58840c4165b68c2f114ae20263"
I20260808 20:12:39.640919 10645 leader_election.cc:288] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 4 election: Requested vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:39.641104 10532 raft_consensus.cc:3037] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:12:39.644454 10529 raft_consensus.cc:2454] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff9f26e2cbef4d1fbf9ca975e1959190 in term 2.
I20260808 20:12:39.645442 10589 leader_election.cc:302] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7858bb58840c4165b68c2f114ae20263, ff9f26e2cbef4d1fbf9ca975e1959190; no voters:
I20260808 20:12:39.646194 10962 raft_consensus.cc:2781] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:12:39.646579 10962 raft_consensus.cc:686] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 LEADER]: Becoming Leader. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Running, Role: LEADER
I20260808 20:12:39.647122 10532 raft_consensus.cc:2454] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff9f26e2cbef4d1fbf9ca975e1959190 in term 4.
I20260808 20:12:39.647221 10962 consensus_queue.cc:237] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:39.648120 10589 leader_election.cc:302] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7858bb58840c4165b68c2f114ae20263, ff9f26e2cbef4d1fbf9ca975e1959190; no voters:
I20260808 20:12:39.648186 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afafd38c69d4fa5b180f254a38ee912" candidate_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "186c827b8b48447b94beb413077d76fe"
I20260808 20:12:39.648757 10761 raft_consensus.cc:3032] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:12:39.648929 10760 raft_consensus.cc:2454] T 94846b349249412faaa65fb6ef97df95 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff9f26e2cbef4d1fbf9ca975e1959190 in term 2.
I20260808 20:12:39.649188 10998 raft_consensus.cc:2781] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:12:39.649108 10761 raft_consensus.cc:729] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 186c827b8b48447b94beb413077d76fe, State: Running, Role: LEADER
I20260808 20:12:39.649682 10998 raft_consensus.cc:686] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 4 LEADER]: Becoming Leader. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Running, Role: LEADER
I20260808 20:12:39.650333 10761 consensus_queue.cc:260] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:39.650606 10998 consensus_queue.cc:237] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:39.651315 10761 raft_consensus.cc:3037] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:12:39.657228 10389 catalog_manager.cc:5495] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190 reported cstate change: term changed from 1 to 2, leader changed from 7858bb58840c4165b68c2f114ae20263 (127.10.24.1) to ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2). New cstate: current_term: 2 leader_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:39.658053 10761 raft_consensus.cc:2454] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff9f26e2cbef4d1fbf9ca975e1959190 in term 4.
I20260808 20:12:39.667971 10389 catalog_manager.cc:5495] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 reported cstate change: term changed from 3 to 4, leader changed from 186c827b8b48447b94beb413077d76fe (127.10.24.3) to ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2). New cstate: current_term: 4 leader_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:39.734714 10761 raft_consensus.cc:1261] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [term 3 FOLLOWER]: Refusing update from remote peer ff9f26e2cbef4d1fbf9ca975e1959190: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:12:39.735992 10998 consensus_queue.cc:1035] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:12:39.749326 10532 raft_consensus.cc:1261] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [term 3 FOLLOWER]: Refusing update from remote peer ff9f26e2cbef4d1fbf9ca975e1959190: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:12:39.752573 11006 consensus_queue.cc:1035] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:12:39.758021 10761 raft_consensus.cc:1261] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [term 3 FOLLOWER]: Refusing update from remote peer ff9f26e2cbef4d1fbf9ca975e1959190: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:12:39.760530 11023 consensus_queue.cc:1035] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:12:39.775343 10532 raft_consensus.cc:1261] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [term 3 FOLLOWER]: Refusing update from remote peer ff9f26e2cbef4d1fbf9ca975e1959190: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:12:39.776692 11023 consensus_queue.cc:1035] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:12:40.017930 10761 raft_consensus.cc:1261] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 3 FOLLOWER]: Refusing update from remote peer ff9f26e2cbef4d1fbf9ca975e1959190: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:12:40.019203 11023 consensus_queue.cc:1035] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:12:40.032347 10532 raft_consensus.cc:1261] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 3 FOLLOWER]: Refusing update from remote peer ff9f26e2cbef4d1fbf9ca975e1959190: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:12:40.033610 11023 consensus_queue.cc:1035] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:12:40.137210 10761 raft_consensus.cc:1261] T 94846b349249412faaa65fb6ef97df95 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Refusing update from remote peer ff9f26e2cbef4d1fbf9ca975e1959190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:40.138535 11023 consensus_queue.cc:1035] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:40.147018 10532 raft_consensus.cc:1261] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Refusing update from remote peer ff9f26e2cbef4d1fbf9ca975e1959190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:40.148499 10998 consensus_queue.cc:1035] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:40.215037 10532 raft_consensus.cc:1261] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [term 4 FOLLOWER]: Refusing update from remote peer ff9f26e2cbef4d1fbf9ca975e1959190: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 20:12:40.216228 11023 consensus_queue.cc:1035] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:12:40.220657 10760 raft_consensus.cc:1261] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 4 FOLLOWER]: Refusing update from remote peer ff9f26e2cbef4d1fbf9ca975e1959190: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 20:12:40.222110 10962 consensus_queue.cc:1035] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:12:42.281654 10336 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20260808 20:12:42.282320 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:12:42.288792 10645 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10c2df6974bc48a19899f8ef6a1932dc"
dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
mode: GRACEFUL
new_leader_uuid: "7858bb58840c4165b68c2f114ae20263"
from {username='slave'} at 127.0.0.1:38460
I20260808 20:12:42.289825 10645 raft_consensus.cc:595] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 LEADER]: Received request to transfer leadership to TS 7858bb58840c4165b68c2f114ae20263
I20260808 20:12:42.292173 10645 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3afafd38c69d4fa5b180f254a38ee912"
dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
mode: GRACEFUL
new_leader_uuid: "7858bb58840c4165b68c2f114ae20263"
from {username='slave'} at 127.0.0.1:38460
I20260808 20:12:42.292600 10645 raft_consensus.cc:595] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 4 LEADER]: Received request to transfer leadership to TS 7858bb58840c4165b68c2f114ae20263
I20260808 20:12:42.294479 10645 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4c8755efc8014e31a8f711fd137edef0"
dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
mode: GRACEFUL
new_leader_uuid: "186c827b8b48447b94beb413077d76fe"
from {username='slave'} at 127.0.0.1:38460
I20260808 20:12:42.294874 10645 raft_consensus.cc:595] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 LEADER]: Received request to transfer leadership to TS 186c827b8b48447b94beb413077d76fe
I20260808 20:12:42.296639 10645 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b99f754163dc41dbb325bba1ba45c63e"
dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
mode: GRACEFUL
new_leader_uuid: "186c827b8b48447b94beb413077d76fe"
from {username='slave'} at 127.0.0.1:38460
I20260808 20:12:42.297031 10645 raft_consensus.cc:595] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 LEADER]: Received request to transfer leadership to TS 186c827b8b48447b94beb413077d76fe
I20260808 20:12:42.298746 10645 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e55ac233626f41e5b6ef9aa8f8204df8"
dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
mode: GRACEFUL
new_leader_uuid: "7858bb58840c4165b68c2f114ae20263"
from {username='slave'} at 127.0.0.1:38460
I20260808 20:12:42.299158 10645 raft_consensus.cc:595] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 LEADER]: Received request to transfer leadership to TS 7858bb58840c4165b68c2f114ae20263
I20260808 20:12:42.299922 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260808 20:12:42.300279 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:12:42.367300 11046 raft_consensus.cc:979] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190: : Instructing follower 186c827b8b48447b94beb413077d76fe to start an election
I20260808 20:12:42.367794 11023 raft_consensus.cc:1067] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 LEADER]: Signalling peer 186c827b8b48447b94beb413077d76fe to start an election
I20260808 20:12:42.369165 10761 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4c8755efc8014e31a8f711fd137edef0"
dest_uuid: "186c827b8b48447b94beb413077d76fe"
from {username='slave'} at 127.0.0.1:58148
I20260808 20:12:42.369607 10761 raft_consensus.cc:482] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:42.369918 10761 raft_consensus.cc:3037] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:12:42.375438 10761 raft_consensus.cc:504] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:42.377476 10761 leader_election.cc:288] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 3 election: Requested vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407)
I20260808 20:12:42.378329 10529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8755efc8014e31a8f711fd137edef0" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7858bb58840c4165b68c2f114ae20263"
I20260808 20:12:42.378386 10645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8755efc8014e31a8f711fd137edef0" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
I20260808 20:12:42.379026 10529 raft_consensus.cc:3037] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:12:42.379092 10645 raft_consensus.cc:3032] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:12:42.379443 10645 raft_consensus.cc:729] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Running, Role: LEADER
I20260808 20:12:42.379990 10645 consensus_queue.cc:260] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:42.380995 10645 raft_consensus.cc:3037] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:12:42.383339 10529 raft_consensus.cc:2454] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186c827b8b48447b94beb413077d76fe in term 3.
I20260808 20:12:42.384366 10705 leader_election.cc:302] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, 7858bb58840c4165b68c2f114ae20263; no voters:
I20260808 20:12:42.385257 11038 raft_consensus.cc:2781] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:12:42.385895 10645 raft_consensus.cc:2454] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186c827b8b48447b94beb413077d76fe in term 3.
I20260808 20:12:42.386243 11038 raft_consensus.cc:686] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe [term 3 LEADER]: Becoming Leader. State: Replica: 186c827b8b48447b94beb413077d76fe, State: Running, Role: LEADER
I20260808 20:12:42.387508 11038 consensus_queue.cc:237] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:42.393568 10385 catalog_manager.cc:5495] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe reported cstate change: term changed from 2 to 3, leader changed from ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2) to 186c827b8b48447b94beb413077d76fe (127.10.24.3). New cstate: current_term: 3 leader_uuid: "186c827b8b48447b94beb413077d76fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: HEALTHY } } }
I20260808 20:12:42.411523 11046 raft_consensus.cc:979] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190: : Instructing follower 7858bb58840c4165b68c2f114ae20263 to start an election
I20260808 20:12:42.411863 11023 raft_consensus.cc:1067] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 LEADER]: Signalling peer 7858bb58840c4165b68c2f114ae20263 to start an election
I20260808 20:12:42.413066 10529 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10c2df6974bc48a19899f8ef6a1932dc"
dest_uuid: "7858bb58840c4165b68c2f114ae20263"
from {username='slave'} at 127.0.0.1:52520
I20260808 20:12:42.413425 10529 raft_consensus.cc:482] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:42.413651 10529 raft_consensus.cc:3037] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:12:42.417533 10529 raft_consensus.cc:504] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:42.419188 10529 leader_election.cc:288] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 4 election: Requested vote from peers ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:42.419839 10645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c2df6974bc48a19899f8ef6a1932dc" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
I20260808 20:12:42.420331 10645 raft_consensus.cc:3032] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:12:42.420279 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c2df6974bc48a19899f8ef6a1932dc" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "186c827b8b48447b94beb413077d76fe"
I20260808 20:12:42.420656 10645 raft_consensus.cc:729] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Running, Role: LEADER
I20260808 20:12:42.420819 10761 raft_consensus.cc:3037] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:12:42.421367 10645 consensus_queue.cc:260] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:42.422379 10645 raft_consensus.cc:3037] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:12:42.424715 10761 raft_consensus.cc:2454] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7858bb58840c4165b68c2f114ae20263 in term 4.
I20260808 20:12:42.425715 10480 leader_election.cc:302] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, 7858bb58840c4165b68c2f114ae20263; no voters:
I20260808 20:12:42.427047 10645 raft_consensus.cc:2454] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7858bb58840c4165b68c2f114ae20263 in term 4.
I20260808 20:12:42.427424 11064 raft_consensus.cc:2781] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:12:42.428550 11064 raft_consensus.cc:686] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 4 LEADER]: Becoming Leader. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Running, Role: LEADER
I20260808 20:12:42.429313 11064 consensus_queue.cc:237] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:42.435874 10385 catalog_manager.cc:5495] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 reported cstate change: term changed from 3 to 4, leader changed from ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2) to 7858bb58840c4165b68c2f114ae20263 (127.10.24.1). New cstate: current_term: 4 leader_uuid: "7858bb58840c4165b68c2f114ae20263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:42.477483 11046 raft_consensus.cc:979] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190: : Instructing follower 186c827b8b48447b94beb413077d76fe to start an election
I20260808 20:12:42.477905 11023 raft_consensus.cc:1067] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 LEADER]: Signalling peer 186c827b8b48447b94beb413077d76fe to start an election
I20260808 20:12:42.479220 10761 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b99f754163dc41dbb325bba1ba45c63e"
dest_uuid: "186c827b8b48447b94beb413077d76fe"
from {username='slave'} at 127.0.0.1:58148
I20260808 20:12:42.479709 10761 raft_consensus.cc:482] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:42.480018 10761 raft_consensus.cc:3037] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:42.484164 10761 raft_consensus.cc:504] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:42.485496 10761 leader_election.cc:288] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 2 election: Requested vote from peers 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783), ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407)
I20260808 20:12:42.486245 10529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99f754163dc41dbb325bba1ba45c63e" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7858bb58840c4165b68c2f114ae20263"
I20260808 20:12:42.486773 10529 raft_consensus.cc:3037] T b99f754163dc41dbb325bba1ba45c63e P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:42.487597 10645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99f754163dc41dbb325bba1ba45c63e" candidate_uuid: "186c827b8b48447b94beb413077d76fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
I20260808 20:12:42.488147 10645 raft_consensus.cc:3032] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:12:42.488443 10645 raft_consensus.cc:729] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Running, Role: LEADER
I20260808 20:12:42.489099 10645 consensus_queue.cc:260] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:42.489951 10645 raft_consensus.cc:3037] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:42.490708 10529 raft_consensus.cc:2454] T b99f754163dc41dbb325bba1ba45c63e P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186c827b8b48447b94beb413077d76fe in term 2.
I20260808 20:12:42.491451 10705 leader_election.cc:302] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, 7858bb58840c4165b68c2f114ae20263; no voters:
I20260808 20:12:42.492058 11038 raft_consensus.cc:2781] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:12:42.492399 11038 raft_consensus.cc:686] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe [term 2 LEADER]: Becoming Leader. State: Replica: 186c827b8b48447b94beb413077d76fe, State: Running, Role: LEADER
I20260808 20:12:42.493124 11038 consensus_queue.cc:237] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:42.494094 10645 raft_consensus.cc:2454] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186c827b8b48447b94beb413077d76fe in term 2.
I20260808 20:12:42.498941 10385 catalog_manager.cc:5495] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe reported cstate change: term changed from 1 to 2, leader changed from ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2) to 186c827b8b48447b94beb413077d76fe (127.10.24.3). New cstate: current_term: 2 leader_uuid: "186c827b8b48447b94beb413077d76fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: HEALTHY } } }
I20260808 20:12:42.536254 11046 raft_consensus.cc:979] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190: : Instructing follower 7858bb58840c4165b68c2f114ae20263 to start an election
I20260808 20:12:42.536669 11023 raft_consensus.cc:1067] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 4 LEADER]: Signalling peer 7858bb58840c4165b68c2f114ae20263 to start an election
I20260808 20:12:42.537832 10529 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3afafd38c69d4fa5b180f254a38ee912"
dest_uuid: "7858bb58840c4165b68c2f114ae20263"
from {username='slave'} at 127.0.0.1:52520
I20260808 20:12:42.538213 10529 raft_consensus.cc:482] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:42.538442 10529 raft_consensus.cc:3037] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:12:42.542233 10529 raft_consensus.cc:504] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:42.543547 10529 leader_election.cc:288] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 5 election: Requested vote from peers ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:42.544196 10645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afafd38c69d4fa5b180f254a38ee912" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
I20260808 20:12:42.544421 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afafd38c69d4fa5b180f254a38ee912" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "186c827b8b48447b94beb413077d76fe"
I20260808 20:12:42.544646 10645 raft_consensus.cc:3032] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:12:42.544864 10761 raft_consensus.cc:3037] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:12:42.544931 10645 raft_consensus.cc:729] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Running, Role: LEADER
I20260808 20:12:42.545608 10645 consensus_queue.cc:260] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:42.546510 10645 raft_consensus.cc:3037] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:12:42.548766 10761 raft_consensus.cc:2454] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7858bb58840c4165b68c2f114ae20263 in term 5.
I20260808 20:12:42.549595 10480 leader_election.cc:302] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, 7858bb58840c4165b68c2f114ae20263; no voters:
I20260808 20:12:42.550321 11064 raft_consensus.cc:2781] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:12:42.550679 11064 raft_consensus.cc:686] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [term 5 LEADER]: Becoming Leader. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Running, Role: LEADER
I20260808 20:12:42.550956 10645 raft_consensus.cc:2454] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7858bb58840c4165b68c2f114ae20263 in term 5.
I20260808 20:12:42.551461 11064 consensus_queue.cc:237] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:42.557966 10385 catalog_manager.cc:5495] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 reported cstate change: term changed from 4 to 5, leader changed from ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2) to 7858bb58840c4165b68c2f114ae20263 (127.10.24.1). New cstate: current_term: 5 leader_uuid: "7858bb58840c4165b68c2f114ae20263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:42.562781 11046 raft_consensus.cc:979] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190: : Instructing follower 7858bb58840c4165b68c2f114ae20263 to start an election
I20260808 20:12:42.563138 11046 raft_consensus.cc:1067] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 LEADER]: Signalling peer 7858bb58840c4165b68c2f114ae20263 to start an election
I20260808 20:12:42.564405 10529 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e55ac233626f41e5b6ef9aa8f8204df8"
dest_uuid: "7858bb58840c4165b68c2f114ae20263"
from {username='slave'} at 127.0.0.1:52520
I20260808 20:12:42.565207 10529 raft_consensus.cc:482] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:42.565515 10529 raft_consensus.cc:3037] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:42.569423 10529 raft_consensus.cc:504] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:42.570576 10529 leader_election.cc:288] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 2 election: Requested vote from peers ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407), 186c827b8b48447b94beb413077d76fe (127.10.24.3:37651)
I20260808 20:12:42.571254 10645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55ac233626f41e5b6ef9aa8f8204df8" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190"
I20260808 20:12:42.571419 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55ac233626f41e5b6ef9aa8f8204df8" candidate_uuid: "7858bb58840c4165b68c2f114ae20263" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "186c827b8b48447b94beb413077d76fe"
I20260808 20:12:42.571764 10645 raft_consensus.cc:3032] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:12:42.571871 10761 raft_consensus.cc:3037] T e55ac233626f41e5b6ef9aa8f8204df8 P 186c827b8b48447b94beb413077d76fe [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:42.572053 10645 raft_consensus.cc:729] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff9f26e2cbef4d1fbf9ca975e1959190, State: Running, Role: LEADER
I20260808 20:12:42.572751 10645 consensus_queue.cc:260] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:42.573853 10645 raft_consensus.cc:3037] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:42.577248 10761 raft_consensus.cc:2454] T e55ac233626f41e5b6ef9aa8f8204df8 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7858bb58840c4165b68c2f114ae20263 in term 2.
I20260808 20:12:42.578145 10480 leader_election.cc:302] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186c827b8b48447b94beb413077d76fe, 7858bb58840c4165b68c2f114ae20263; no voters:
I20260808 20:12:42.578737 11064 raft_consensus.cc:2781] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:12:42.578872 10645 raft_consensus.cc:2454] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7858bb58840c4165b68c2f114ae20263 in term 2.
I20260808 20:12:42.579092 11064 raft_consensus.cc:686] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [term 2 LEADER]: Becoming Leader. State: Replica: 7858bb58840c4165b68c2f114ae20263, State: Running, Role: LEADER
I20260808 20:12:42.579927 11064 consensus_queue.cc:237] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } }
I20260808 20:12:42.586071 10385 catalog_manager.cc:5495] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 reported cstate change: term changed from 1 to 2, leader changed from ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2) to 7858bb58840c4165b68c2f114ae20263 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "7858bb58840c4165b68c2f114ae20263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:42.888836 10761 raft_consensus.cc:1261] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 4 FOLLOWER]: Refusing update from remote peer 7858bb58840c4165b68c2f114ae20263: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 20:12:42.889849 11064 consensus_queue.cc:1035] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:12:42.901499 10645 raft_consensus.cc:1261] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [term 4 FOLLOWER]: Refusing update from remote peer 7858bb58840c4165b68c2f114ae20263: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 20:12:42.902922 11066 consensus_queue.cc:1035] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 20:12:42.929625 10645 raft_consensus.cc:1261] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Refusing update from remote peer 186c827b8b48447b94beb413077d76fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:42.930562 11038 consensus_queue.cc:1035] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:42.932041 10645 raft_consensus.cc:1261] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 FOLLOWER]: Refusing update from remote peer 186c827b8b48447b94beb413077d76fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:12:42.933211 11038 consensus_queue.cc:1035] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:12:42.938700 10529 raft_consensus.cc:1261] T b99f754163dc41dbb325bba1ba45c63e P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Refusing update from remote peer 186c827b8b48447b94beb413077d76fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:42.940263 11047 consensus_queue.cc:1035] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:42.945057 10532 raft_consensus.cc:1261] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [term 3 FOLLOWER]: Refusing update from remote peer 186c827b8b48447b94beb413077d76fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:12:42.946512 11047 consensus_queue.cc:1035] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "7858bb58840c4165b68c2f114ae20263" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:12:42.962073 10645 raft_consensus.cc:1261] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 5 FOLLOWER]: Refusing update from remote peer 7858bb58840c4165b68c2f114ae20263: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260808 20:12:42.963063 11066 consensus_queue.cc:1035] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:12:42.971812 10761 raft_consensus.cc:1261] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 5 FOLLOWER]: Refusing update from remote peer 7858bb58840c4165b68c2f114ae20263: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260808 20:12:42.973165 11085 consensus_queue.cc:1035] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:12:43.000352 10645 raft_consensus.cc:1261] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Refusing update from remote peer 7858bb58840c4165b68c2f114ae20263: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:43.001223 11085 consensus_queue.cc:1035] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9f26e2cbef4d1fbf9ca975e1959190" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:43.010048 10760 raft_consensus.cc:1261] T e55ac233626f41e5b6ef9aa8f8204df8 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Refusing update from remote peer 7858bb58840c4165b68c2f114ae20263: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:43.011343 11064 consensus_queue.cc:1035] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186c827b8b48447b94beb413077d76fe" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:43.301062 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:12:43.306257 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:12:43.306615 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:12:44.307381 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:12:44.312295 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:12:44.312650 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:12:45.313424 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:12:45.319018 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:12:45.319448 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:12:46.320196 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:12:46.325524 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:12:46.325872 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:12:47.326601 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:12:47.331138 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:12:47.331485 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:12:48.332221 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:12:48.336707 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:12:48.337067 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:12:49.337828 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:12:49.342484 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:12:49.342849 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:12:50.343595 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:12:50.348129 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:12:50.348487 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:12:51.349236 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:12:51.353770 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:12:51.354135 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:12:52.379587 10336 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20260808 20:12:52.380168 10336 tablet_server.cc:178] TabletServer@127.10.24.1:0 shutting down...
I20260808 20:12:52.405012 10336 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260808 20:12:52.405656 10336 tablet_replica.cc:330] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263: stopping tablet replica
I20260808 20:12:52.406386 10336 raft_consensus.cc:2229] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 20:12:52.406980 10336 raft_consensus.cc:2258] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.409686 10336 tablet_replica.cc:330] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263: stopping tablet replica
I20260808 20:12:52.410336 10336 raft_consensus.cc:2229] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 20:12:52.410809 10336 raft_consensus.cc:2258] T 4c8755efc8014e31a8f711fd137edef0 P 7858bb58840c4165b68c2f114ae20263 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.412700 10336 tablet_replica.cc:330] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263: stopping tablet replica
I20260808 20:12:52.413182 10336 raft_consensus.cc:2229] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [term 5 LEADER]: Raft consensus shutting down.
I20260808 20:12:52.413913 10336 raft_consensus.cc:2258] T 3afafd38c69d4fa5b180f254a38ee912 P 7858bb58840c4165b68c2f114ae20263 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.415403 10336 tablet_replica.cc:330] T b99f754163dc41dbb325bba1ba45c63e P 7858bb58840c4165b68c2f114ae20263: stopping tablet replica
I20260808 20:12:52.415846 10336 raft_consensus.cc:2229] T b99f754163dc41dbb325bba1ba45c63e P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 20:12:52.416213 10336 raft_consensus.cc:2258] T b99f754163dc41dbb325bba1ba45c63e P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.417562 10336 tablet_replica.cc:330] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263: stopping tablet replica
I20260808 20:12:52.417985 10336 raft_consensus.cc:2229] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [term 2 LEADER]: Raft consensus shutting down.
I20260808 20:12:52.418674 10336 raft_consensus.cc:2258] T e55ac233626f41e5b6ef9aa8f8204df8 P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.420109 10336 tablet_replica.cc:330] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263: stopping tablet replica
I20260808 20:12:52.420554 10336 raft_consensus.cc:2229] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 4 LEADER]: Raft consensus shutting down.
I20260808 20:12:52.421239 10336 raft_consensus.cc:2258] T 10c2df6974bc48a19899f8ef6a1932dc P 7858bb58840c4165b68c2f114ae20263 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.422844 10336 tablet_replica.cc:330] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263: stopping tablet replica
I20260808 20:12:52.423285 10336 raft_consensus.cc:2229] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 20:12:52.423684 10336 raft_consensus.cc:2258] T 2a552acaffcd45268f344a1aa35cb132 P 7858bb58840c4165b68c2f114ae20263 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.425277 10336 tablet_replica.cc:330] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263: stopping tablet replica
I20260808 20:12:52.425681 10336 raft_consensus.cc:2229] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 20:12:52.426034 10336 raft_consensus.cc:2258] T 94846b349249412faaa65fb6ef97df95 P 7858bb58840c4165b68c2f114ae20263 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.427527 10336 tablet_replica.cc:330] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263: stopping tablet replica
I20260808 20:12:52.427918 10336 raft_consensus.cc:2229] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 20:12:52.428275 10336 raft_consensus.cc:2258] T 7b193cc24b354319ab91439036f51881 P 7858bb58840c4165b68c2f114ae20263 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260808 20:12:52.582886 10705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.24.1:39783: connect: Connection refused (error 111)
I20260808 20:12:52.585690 10336 tablet_server.cc:194] TabletServer@127.10.24.1:0 shutdown complete.
W20260808 20:12:52.587132 10589 consensus_peers.cc:487] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 -> Peer 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783): Couldn't send request to peer 7858bb58840c4165b68c2f114ae20263. Status: Network error: Client connection negotiation failed: client connection to 127.10.24.1:39783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 20:12:52.587689 10589 consensus_peers.cc:487] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 -> Peer 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783): Couldn't send request to peer 7858bb58840c4165b68c2f114ae20263. Status: Network error: Client connection negotiation failed: client connection to 127.10.24.1:39783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 20:12:52.587968 10705 consensus_peers.cc:487] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe -> Peer 7858bb58840c4165b68c2f114ae20263 (127.10.24.1:39783): Couldn't send request to peer 7858bb58840c4165b68c2f114ae20263. Status: Network error: Client connection negotiation failed: client connection to 127.10.24.1:39783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 20:12:52.605805 10336 tablet_server.cc:178] TabletServer@127.10.24.2:0 shutting down...
I20260808 20:12:52.631399 10336 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260808 20:12:52.632048 10336 tablet_replica.cc:330] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190: stopping tablet replica
I20260808 20:12:52.632717 10336 raft_consensus.cc:2229] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 20:12:52.633273 10336 raft_consensus.cc:2258] T 4c8755efc8014e31a8f711fd137edef0 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.635336 10336 tablet_replica.cc:330] T 00c98a0a1ac84a6aa4b7431b491e4e52 P ff9f26e2cbef4d1fbf9ca975e1959190: stopping tablet replica
I20260808 20:12:52.635995 10336 raft_consensus.cc:2229] T 00c98a0a1ac84a6aa4b7431b491e4e52 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 20:12:52.636602 10336 raft_consensus.cc:2258] T 00c98a0a1ac84a6aa4b7431b491e4e52 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260808 20:12:52.638798 10704 consensus_peers.cc:487] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe -> Peer ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407): Couldn't send request to peer ff9f26e2cbef4d1fbf9ca975e1959190. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:12:52.639024 10336 tablet_replica.cc:330] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190: stopping tablet replica
I20260808 20:12:52.639621 10336 raft_consensus.cc:2229] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 LEADER]: Raft consensus shutting down.
I20260808 20:12:52.640588 10336 raft_consensus.cc:2258] T 7b193cc24b354319ab91439036f51881 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260808 20:12:52.642017 10704 consensus_peers.cc:487] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe -> Peer ff9f26e2cbef4d1fbf9ca975e1959190 (127.10.24.2:43407): Couldn't send request to peer ff9f26e2cbef4d1fbf9ca975e1959190. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:12:52.642642 10336 tablet_replica.cc:330] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190: stopping tablet replica
I20260808 20:12:52.643143 10336 raft_consensus.cc:2229] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 LEADER]: Raft consensus shutting down.
I20260808 20:12:52.643846 10336 raft_consensus.cc:2258] T 2a552acaffcd45268f344a1aa35cb132 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.645516 10336 tablet_replica.cc:330] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190: stopping tablet replica
I20260808 20:12:52.645982 10336 raft_consensus.cc:2229] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 20:12:52.646397 10336 raft_consensus.cc:2258] T 10c2df6974bc48a19899f8ef6a1932dc P ff9f26e2cbef4d1fbf9ca975e1959190 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.647986 10336 tablet_replica.cc:330] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190: stopping tablet replica
I20260808 20:12:52.648427 10336 raft_consensus.cc:2229] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 20:12:52.648802 10336 raft_consensus.cc:2258] T b99f754163dc41dbb325bba1ba45c63e P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.650403 10336 tablet_replica.cc:330] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190: stopping tablet replica
I20260808 20:12:52.650862 10336 raft_consensus.cc:2229] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260808 20:12:52.651239 10336 raft_consensus.cc:2258] T 3afafd38c69d4fa5b180f254a38ee912 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.652699 10336 tablet_replica.cc:330] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190: stopping tablet replica
I20260808 20:12:52.653178 10336 raft_consensus.cc:2229] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 20:12:52.653551 10336 raft_consensus.cc:2258] T e55ac233626f41e5b6ef9aa8f8204df8 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.655025 10336 tablet_replica.cc:330] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190: stopping tablet replica
I20260808 20:12:52.655386 10336 raft_consensus.cc:2229] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 LEADER]: Raft consensus shutting down.
I20260808 20:12:52.656054 10336 raft_consensus.cc:2258] T 94846b349249412faaa65fb6ef97df95 P ff9f26e2cbef4d1fbf9ca975e1959190 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.682642 10336 tablet_server.cc:194] TabletServer@127.10.24.2:0 shutdown complete.
I20260808 20:12:52.698743 10336 tablet_server.cc:178] TabletServer@127.10.24.3:0 shutting down...
I20260808 20:12:52.720686 10336 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260808 20:12:52.721366 10336 tablet_replica.cc:330] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe: stopping tablet replica
I20260808 20:12:52.721946 10336 raft_consensus.cc:2229] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe [term 2 LEADER]: Raft consensus shutting down.
I20260808 20:12:52.722649 10336 raft_consensus.cc:2258] T 00c98a0a1ac84a6aa4b7431b491e4e52 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.724231 10336 tablet_replica.cc:330] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe: stopping tablet replica
I20260808 20:12:52.724695 10336 raft_consensus.cc:2229] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe [term 3 LEADER]: Raft consensus shutting down.
I20260808 20:12:52.725406 10336 raft_consensus.cc:2258] T 4c8755efc8014e31a8f711fd137edef0 P 186c827b8b48447b94beb413077d76fe [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.726913 10336 tablet_replica.cc:330] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe: stopping tablet replica
I20260808 20:12:52.727499 10336 raft_consensus.cc:2229] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe [term 2 LEADER]: Raft consensus shutting down.
I20260808 20:12:52.728243 10336 raft_consensus.cc:2258] T b99f754163dc41dbb325bba1ba45c63e P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.729715 10336 tablet_replica.cc:330] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe: stopping tablet replica
I20260808 20:12:52.730122 10336 raft_consensus.cc:2229] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 5 FOLLOWER]: Raft consensus shutting down.
I20260808 20:12:52.730499 10336 raft_consensus.cc:2258] T 3afafd38c69d4fa5b180f254a38ee912 P 186c827b8b48447b94beb413077d76fe [term 5 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.731889 10336 tablet_replica.cc:330] T e55ac233626f41e5b6ef9aa8f8204df8 P 186c827b8b48447b94beb413077d76fe: stopping tablet replica
I20260808 20:12:52.732285 10336 raft_consensus.cc:2229] T e55ac233626f41e5b6ef9aa8f8204df8 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 20:12:52.732643 10336 raft_consensus.cc:2258] T e55ac233626f41e5b6ef9aa8f8204df8 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.734064 10336 tablet_replica.cc:330] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe: stopping tablet replica
I20260808 20:12:52.734464 10336 raft_consensus.cc:2229] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 20:12:52.734860 10336 raft_consensus.cc:2258] T 10c2df6974bc48a19899f8ef6a1932dc P 186c827b8b48447b94beb413077d76fe [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.736311 10336 tablet_replica.cc:330] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe: stopping tablet replica
I20260808 20:12:52.736703 10336 raft_consensus.cc:2229] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 20:12:52.737118 10336 raft_consensus.cc:2258] T 2a552acaffcd45268f344a1aa35cb132 P 186c827b8b48447b94beb413077d76fe [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.738493 10336 tablet_replica.cc:330] T 94846b349249412faaa65fb6ef97df95 P 186c827b8b48447b94beb413077d76fe: stopping tablet replica
I20260808 20:12:52.738871 10336 raft_consensus.cc:2229] T 94846b349249412faaa65fb6ef97df95 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 20:12:52.739188 10336 raft_consensus.cc:2258] T 94846b349249412faaa65fb6ef97df95 P 186c827b8b48447b94beb413077d76fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.740494 10336 tablet_replica.cc:330] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe: stopping tablet replica
I20260808 20:12:52.740852 10336 raft_consensus.cc:2229] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 20:12:52.741245 10336 raft_consensus.cc:2258] T 7b193cc24b354319ab91439036f51881 P 186c827b8b48447b94beb413077d76fe [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:52.781643 10336 tablet_server.cc:194] TabletServer@127.10.24.3:0 shutdown complete.
I20260808 20:12:52.797955 10336 master.cc:537] Master@127.10.24.62:33591 shutting down...
I20260808 20:12:53.029850 10336 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c [term 1 LEADER]: Raft consensus shutting down.
I20260808 20:12:53.030598 10336 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:12:53.030942 10336 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7960e0673c6f456db25b0283a3a3732c: stopping tablet replica
I20260808 20:12:53.048707 10336 master.cc:552] Master@127.10.24.62:33591 shutdown complete.
186c827b8b48447b94beb413077d76fe 1
7858bb58840c4165b68c2f114ae20263 4
ff9f26e2cbef4d1fbf9ca975e1959190 4
186c827b8b48447b94beb413077d76fe 3
7858bb58840c4165b68c2f114ae20263 3
ff9f26e2cbef4d1fbf9ca975e1959190 3
186c827b8b48447b94beb413077d76fe 1
7858bb58840c4165b68c2f114ae20263 0
ff9f26e2cbef4d1fbf9ca975e1959190 8
186c827b8b48447b94beb413077d76fe 3
7858bb58840c4165b68c2f114ae20263 3
ff9f26e2cbef4d1fbf9ca975e1959190 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (31364 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260808 20:12:53.085438 10336 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.10.24.62:46075
I20260808 20:12:53.086457 10336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 20:12:53.091753 11121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 20:12:53.092058 11124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 20:12:53.093116 11127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 20:12:53.094542 10336 server_base.cc:1034] running on GCE node
I20260808 20:12:53.095427 10336 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 20:12:53.095605 10336 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 20:12:53.095710 10336 hybrid_clock.cc:648] HybridClock initialized: now 1786219973095703 us; error 0 us; skew 500 ppm
I20260808 20:12:53.096134 10336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 20:12:53.098389 10336 webserver.cc:460] Webserver started at http://127.10.24.62:43811/ using document root <none> and password file <none>
I20260808 20:12:53.098809 10336 fs_manager.cc:359] Metadata directory not provided
I20260808 20:12:53.098958 10336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 20:12:53.099200 10336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 20:12:53.100481 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786219941176655-10336-0/minicluster-data/master-0-root/instance:
uuid: "4b6c1038515743ca968bf5357b32af89"
format_stamp: "Formatted at 2026-08-08 20:12:53 on dist-test-slave-ncp9"
I20260808 20:12:53.104650 10336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260808 20:12:53.107622 11134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 20:12:53.108243 10336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 20:12:53.108484 10336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786219941176655-10336-0/minicluster-data/master-0-root
uuid: "4b6c1038515743ca968bf5357b32af89"
format_stamp: "Formatted at 2026-08-08 20:12:53 on dist-test-slave-ncp9"
I20260808 20:12:53.108737 10336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786219941176655-10336-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786219941176655-10336-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786219941176655-10336-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 20:12:53.132014 10336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 20:12:53.133195 10336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 20:12:53.168825 10336 rpc_server.cc:307] RPC server started. Bound to: 127.10.24.62:46075
I20260808 20:12:53.168928 11206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.24.62:46075 every 8 connection(s)
I20260808 20:12:53.172396 11207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.181137 11207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89: Bootstrap starting.
I20260808 20:12:53.185307 11207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:53.189301 11207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89: No bootstrap required, opened a new log
I20260808 20:12:53.191581 11207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6c1038515743ca968bf5357b32af89" member_type: VOTER }
I20260808 20:12:53.192122 11207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:53.192346 11207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b6c1038515743ca968bf5357b32af89, State: Initialized, Role: FOLLOWER
I20260808 20:12:53.192845 11207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6c1038515743ca968bf5357b32af89" member_type: VOTER }
I20260808 20:12:53.193324 11207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 20:12:53.193540 11207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 20:12:53.193804 11207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:53.197309 11207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6c1038515743ca968bf5357b32af89" member_type: VOTER }
I20260808 20:12:53.197767 11207 leader_election.cc:302] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b6c1038515743ca968bf5357b32af89; no voters:
I20260808 20:12:53.198776 11207 leader_election.cc:288] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 20:12:53.199119 11210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:12:53.200343 11210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89 [term 1 LEADER]: Becoming Leader. State: Replica: 4b6c1038515743ca968bf5357b32af89, State: Running, Role: LEADER
I20260808 20:12:53.200881 11210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6c1038515743ca968bf5357b32af89" member_type: VOTER }
I20260808 20:12:53.201491 11207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 20:12:53.205121 11212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b6c1038515743ca968bf5357b32af89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6c1038515743ca968bf5357b32af89" member_type: VOTER } }
I20260808 20:12:53.205286 11213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b6c1038515743ca968bf5357b32af89. Latest consensus state: current_term: 1 leader_uuid: "4b6c1038515743ca968bf5357b32af89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6c1038515743ca968bf5357b32af89" member_type: VOTER } }
I20260808 20:12:53.205746 11212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89 [sys.catalog]: This master's current role is: LEADER
I20260808 20:12:53.205967 11213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89 [sys.catalog]: This master's current role is: LEADER
I20260808 20:12:53.210547 11220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 20:12:53.215185 11220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 20:12:53.218578 10336 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260808 20:12:53.223342 11220 catalog_manager.cc:1347] Generated new cluster ID: f9b9a7db398f4151adb177ad9a348c75
I20260808 20:12:53.223528 11220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 20:12:53.240876 11220 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 20:12:53.242318 11220 catalog_manager.cc:1504] Loading token signing keys...
I20260808 20:12:53.253232 11220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89: Generated new TSK 0
I20260808 20:12:53.253748 11220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 20:12:53.284600 10336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 20:12:53.290112 11240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 20:12:53.291067 11241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 20:12:53.293391 11243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 20:12:53.293501 10336 server_base.cc:1034] running on GCE node
I20260808 20:12:53.294482 10336 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 20:12:53.294658 10336 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 20:12:53.294819 10336 hybrid_clock.cc:648] HybridClock initialized: now 1786219973294803 us; error 0 us; skew 500 ppm
I20260808 20:12:53.295305 10336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 20:12:53.297504 10336 webserver.cc:460] Webserver started at http://127.10.24.1:36423/ using document root <none> and password file <none>
I20260808 20:12:53.297972 10336 fs_manager.cc:359] Metadata directory not provided
I20260808 20:12:53.298147 10336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 20:12:53.298377 10336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 20:12:53.299433 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786219941176655-10336-0/minicluster-data/ts-0-root/instance:
uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
format_stamp: "Formatted at 2026-08-08 20:12:53 on dist-test-slave-ncp9"
I20260808 20:12:53.303836 10336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 20:12:53.306746 11252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 20:12:53.307380 10336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 20:12:53.307622 10336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786219941176655-10336-0/minicluster-data/ts-0-root
uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
format_stamp: "Formatted at 2026-08-08 20:12:53 on dist-test-slave-ncp9"
I20260808 20:12:53.307870 10336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786219941176655-10336-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786219941176655-10336-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786219941176655-10336-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 20:12:53.318730 10336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 20:12:53.319783 10336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 20:12:53.321106 10336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 20:12:53.323137 10336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 20:12:53.323309 10336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 20:12:53.323519 10336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 20:12:53.323670 10336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 20:12:53.359836 10336 rpc_server.cc:307] RPC server started. Bound to: 127.10.24.1:40959
I20260808 20:12:53.359921 11345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.24.1:40959 every 8 connection(s)
I20260808 20:12:53.364223 10336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 20:12:53.371321 11351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 20:12:53.373142 11353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 20:12:53.374584 11356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 20:12:53.375171 10336 server_base.cc:1034] running on GCE node
I20260808 20:12:53.376456 10336 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 20:12:53.376641 10336 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 20:12:53.376813 10336 hybrid_clock.cc:648] HybridClock initialized: now 1786219973376796 us; error 0 us; skew 500 ppm
I20260808 20:12:53.377507 10336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 20:12:53.378283 11347 heartbeater.cc:347] Connected to a master server at 127.10.24.62:46075
I20260808 20:12:53.378611 11347 heartbeater.cc:464] Registering TS with master...
I20260808 20:12:53.379253 11347 heartbeater.cc:511] Master 127.10.24.62:46075 requested a full tablet report, sending...
I20260808 20:12:53.380227 10336 webserver.cc:460] Webserver started at http://127.10.24.2:45373/ using document root <none> and password file <none>
I20260808 20:12:53.380792 10336 fs_manager.cc:359] Metadata directory not provided
I20260808 20:12:53.381004 10336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 20:12:53.381279 10336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 20:12:53.381234 11163 ts_manager.cc:194] Registered new tserver with Master: bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959)
I20260808 20:12:53.382484 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786219941176655-10336-0/minicluster-data/ts-1-root/instance:
uuid: "2fa933dd5405492aa709133798d4ba98"
format_stamp: "Formatted at 2026-08-08 20:12:53 on dist-test-slave-ncp9"
I20260808 20:12:53.383173 11163 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34188
I20260808 20:12:53.387013 10336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260808 20:12:53.390161 11362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 20:12:53.390831 10336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 20:12:53.391070 10336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786219941176655-10336-0/minicluster-data/ts-1-root
uuid: "2fa933dd5405492aa709133798d4ba98"
format_stamp: "Formatted at 2026-08-08 20:12:53 on dist-test-slave-ncp9"
I20260808 20:12:53.391327 10336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786219941176655-10336-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786219941176655-10336-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786219941176655-10336-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 20:12:53.407188 10336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 20:12:53.408262 10336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 20:12:53.409730 10336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 20:12:53.411808 10336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 20:12:53.411990 10336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 20:12:53.412199 10336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 20:12:53.412334 10336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 20:12:53.452093 10336 rpc_server.cc:307] RPC server started. Bound to: 127.10.24.2:33075
I20260808 20:12:53.452236 11454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.24.2:33075 every 8 connection(s)
I20260808 20:12:53.456429 10336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 20:12:53.462801 11459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 20:12:53.463742 11460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 20:12:53.468241 11455 heartbeater.cc:347] Connected to a master server at 127.10.24.62:46075
I20260808 20:12:53.468609 11455 heartbeater.cc:464] Registering TS with master...
I20260808 20:12:53.469452 10336 server_base.cc:1034] running on GCE node
I20260808 20:12:53.469455 11455 heartbeater.cc:511] Master 127.10.24.62:46075 requested a full tablet report, sending...
W20260808 20:12:53.470993 11462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 20:12:53.471627 11163 ts_manager.cc:194] Registered new tserver with Master: 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075)
I20260808 20:12:53.471808 10336 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 20:12:53.472051 10336 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 20:12:53.472205 10336 hybrid_clock.cc:648] HybridClock initialized: now 1786219973472191 us; error 0 us; skew 500 ppm
I20260808 20:12:53.472733 10336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 20:12:53.473317 11163 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34204
I20260808 20:12:53.475322 10336 webserver.cc:460] Webserver started at http://127.10.24.3:46627/ using document root <none> and password file <none>
I20260808 20:12:53.475800 10336 fs_manager.cc:359] Metadata directory not provided
I20260808 20:12:53.475973 10336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 20:12:53.476195 10336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 20:12:53.477294 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786219941176655-10336-0/minicluster-data/ts-2-root/instance:
uuid: "124d7af2a136423c9165edf5e240ec12"
format_stamp: "Formatted at 2026-08-08 20:12:53 on dist-test-slave-ncp9"
I20260808 20:12:53.481372 10336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 20:12:53.484206 11470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 20:12:53.484860 10336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 20:12:53.485141 10336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786219941176655-10336-0/minicluster-data/ts-2-root
uuid: "124d7af2a136423c9165edf5e240ec12"
format_stamp: "Formatted at 2026-08-08 20:12:53 on dist-test-slave-ncp9"
I20260808 20:12:53.485388 10336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786219941176655-10336-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786219941176655-10336-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786219941176655-10336-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 20:12:53.500275 10336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 20:12:53.501624 10336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 20:12:53.502923 10336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 20:12:53.504936 10336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 20:12:53.505121 10336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 20:12:53.505343 10336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 20:12:53.505473 10336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 20:12:53.542491 10336 rpc_server.cc:307] RPC server started. Bound to: 127.10.24.3:39783
I20260808 20:12:53.542583 11570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.24.3:39783 every 8 connection(s)
I20260808 20:12:53.556006 11572 heartbeater.cc:347] Connected to a master server at 127.10.24.62:46075
I20260808 20:12:53.556355 11572 heartbeater.cc:464] Registering TS with master...
I20260808 20:12:53.557011 11572 heartbeater.cc:511] Master 127.10.24.62:46075 requested a full tablet report, sending...
I20260808 20:12:53.558703 11163 ts_manager.cc:194] Registered new tserver with Master: 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783)
I20260808 20:12:53.559727 10336 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014342133s
I20260808 20:12:53.560453 11163 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34210
I20260808 20:12:53.562378 10336 test_util.cc:268] Using random seed: -1090269194
I20260808 20:12:53.584410 11163 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34224:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 20:12:53.587325 11163 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 20:12:53.652752 11515 tablet_service.cc:1467] Processing CreateTablet for tablet 5e4e13f27b9d4abbbda7599371e4ad48 (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260808 20:12:53.654245 11515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e4e13f27b9d4abbbda7599371e4ad48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.654143 11506 tablet_service.cc:1467] Processing CreateTablet for tablet 44506b0c529c45efbee5ac3003ef6f42 (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260808 20:12:53.655474 11506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44506b0c529c45efbee5ac3003ef6f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.658034 11403 tablet_service.cc:1467] Processing CreateTablet for tablet 5e4e13f27b9d4abbbda7599371e4ad48 (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260808 20:12:53.659281 11403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e4e13f27b9d4abbbda7599371e4ad48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.659135 11513 tablet_service.cc:1467] Processing CreateTablet for tablet 941577432a4440b981675cc78e482325 (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260808 20:12:53.660374 11513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 941577432a4440b981675cc78e482325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.662036 11401 tablet_service.cc:1467] Processing CreateTablet for tablet d6a5465239d74c75bfae9807d91bccba (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260808 20:12:53.663271 11401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6a5465239d74c75bfae9807d91bccba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.666015 11400 tablet_service.cc:1467] Processing CreateTablet for tablet 941577432a4440b981675cc78e482325 (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260808 20:12:53.666188 11507 tablet_service.cc:1467] Processing CreateTablet for tablet 261b728800574016968c1083c414e27f (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260808 20:12:53.667274 11400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 941577432a4440b981675cc78e482325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.667309 11507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 261b728800574016968c1083c414e27f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.658042 11514 tablet_service.cc:1467] Processing CreateTablet for tablet d6a5465239d74c75bfae9807d91bccba (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260808 20:12:53.668813 11514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6a5465239d74c75bfae9807d91bccba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.669997 11399 tablet_service.cc:1467] Processing CreateTablet for tablet 261b728800574016968c1083c414e27f (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260808 20:12:53.671191 11399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 261b728800574016968c1083c414e27f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.674034 11398 tablet_service.cc:1467] Processing CreateTablet for tablet 44506b0c529c45efbee5ac3003ef6f42 (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260808 20:12:53.675252 11398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44506b0c529c45efbee5ac3003ef6f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.677757 11293 tablet_service.cc:1467] Processing CreateTablet for tablet 5e4e13f27b9d4abbbda7599371e4ad48 (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260808 20:12:53.678951 11293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e4e13f27b9d4abbbda7599371e4ad48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.680994 11284 tablet_service.cc:1467] Processing CreateTablet for tablet 089736c28b834bd5a121f818bcb364d7 (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260808 20:12:53.682227 11284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 089736c28b834bd5a121f818bcb364d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.682015 11392 tablet_service.cc:1467] Processing CreateTablet for tablet 089736c28b834bd5a121f818bcb364d7 (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260808 20:12:53.683240 11392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 089736c28b834bd5a121f818bcb364d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.684088 11283 tablet_service.cc:1467] Processing CreateTablet for tablet 07fe59ac00b24efb8271f8e86a33441b (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260808 20:12:53.685338 11283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07fe59ac00b24efb8271f8e86a33441b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.685083 11391 tablet_service.cc:1467] Processing CreateTablet for tablet 07fe59ac00b24efb8271f8e86a33441b (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260808 20:12:53.686257 11391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07fe59ac00b24efb8271f8e86a33441b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.691002 11291 tablet_service.cc:1467] Processing CreateTablet for tablet d6a5465239d74c75bfae9807d91bccba (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260808 20:12:53.692237 11291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6a5465239d74c75bfae9807d91bccba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.694074 11499 tablet_service.cc:1467] Processing CreateTablet for tablet 07fe59ac00b24efb8271f8e86a33441b (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260808 20:12:53.695333 11499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07fe59ac00b24efb8271f8e86a33441b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.697799 11396 tablet_service.cc:1467] Processing CreateTablet for tablet f2719138be134d1382e8c595a0275d7f (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260808 20:12:53.697916 11286 tablet_service.cc:1467] Processing CreateTablet for tablet f616636ffca943a7a1731ee069940307 (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260808 20:12:53.699080 11286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f616636ffca943a7a1731ee069940307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.701947 11288 tablet_service.cc:1467] Processing CreateTablet for tablet 44506b0c529c45efbee5ac3003ef6f42 (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260808 20:12:53.703171 11288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44506b0c529c45efbee5ac3003ef6f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.705937 11395 tablet_service.cc:1467] Processing CreateTablet for tablet f616636ffca943a7a1731ee069940307 (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260808 20:12:53.711124 11500 tablet_service.cc:1467] Processing CreateTablet for tablet 089736c28b834bd5a121f818bcb364d7 (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260808 20:12:53.711658 11501 tablet_service.cc:1467] Processing CreateTablet for tablet 304a2461fac34933aa3a07f54b875889 (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260808 20:12:53.712496 11500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 089736c28b834bd5a121f818bcb364d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.713943 11289 tablet_service.cc:1467] Processing CreateTablet for tablet 261b728800574016968c1083c414e27f (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260808 20:12:53.715957 11285 tablet_service.cc:1467] Processing CreateTablet for tablet 304a2461fac34933aa3a07f54b875889 (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260808 20:12:53.699067 11396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2719138be134d1382e8c595a0275d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.720325 11504 tablet_service.cc:1467] Processing CreateTablet for tablet f2719138be134d1382e8c595a0275d7f (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260808 20:12:53.721062 11393 tablet_service.cc:1467] Processing CreateTablet for tablet 304a2461fac34933aa3a07f54b875889 (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260808 20:12:53.718873 11285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304a2461fac34933aa3a07f54b875889. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.721583 11503 tablet_service.cc:1467] Processing CreateTablet for tablet f616636ffca943a7a1731ee069940307 (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260808 20:12:53.722836 11290 tablet_service.cc:1467] Processing CreateTablet for tablet 941577432a4440b981675cc78e482325 (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260808 20:12:53.726943 11290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 941577432a4440b981675cc78e482325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.727373 11501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304a2461fac34933aa3a07f54b875889. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.709816 11287 tablet_service.cc:1467] Processing CreateTablet for tablet f2719138be134d1382e8c595a0275d7f (DEFAULT_TABLE table=test-workload [id=36e7d5f4e4d04e0bb185b100a52a29b6]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260808 20:12:53.729997 11287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2719138be134d1382e8c595a0275d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.733202 11503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f616636ffca943a7a1731ee069940307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.733824 11504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2719138be134d1382e8c595a0275d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.734400 11289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 261b728800574016968c1083c414e27f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.740386 11395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f616636ffca943a7a1731ee069940307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.741040 11393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304a2461fac34933aa3a07f54b875889. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:12:53.847344 11593 tablet_bootstrap.cc:492] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8: Bootstrap starting.
I20260808 20:12:53.850333 11592 tablet_bootstrap.cc:492] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12: Bootstrap starting.
I20260808 20:12:53.854364 11593 tablet_bootstrap.cc:654] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:53.857308 11592 tablet_bootstrap.cc:654] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:53.862982 11594 tablet_bootstrap.cc:492] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98: Bootstrap starting.
I20260808 20:12:53.864427 11593 tablet_bootstrap.cc:492] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8: No bootstrap required, opened a new log
I20260808 20:12:53.864774 11592 tablet_bootstrap.cc:492] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12: No bootstrap required, opened a new log
I20260808 20:12:53.864780 11593 ts_tablet_manager.cc:1397] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.005s
I20260808 20:12:53.865329 11592 ts_tablet_manager.cc:1397] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260808 20:12:53.867075 11593 raft_consensus.cc:348] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } } peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } }
I20260808 20:12:53.867596 11593 raft_consensus.cc:374] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:53.867827 11593 raft_consensus.cc:729] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Initialized, Role: FOLLOWER
I20260808 20:12:53.867815 11592 raft_consensus.cc:348] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:53.868669 11592 raft_consensus.cc:374] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:53.868573 11593 consensus_queue.cc:260] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } } peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } }
I20260808 20:12:53.869019 11592 raft_consensus.cc:729] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Initialized, Role: FOLLOWER
I20260808 20:12:53.869690 11592 consensus_queue.cc:260] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:53.870457 11594 tablet_bootstrap.cc:654] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:53.871843 11572 heartbeater.cc:503] Master 127.10.24.62:46075 was elected leader, sending a full tablet report...
I20260808 20:12:53.872196 11592 ts_tablet_manager.cc:1428] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260808 20:12:53.873103 11592 tablet_bootstrap.cc:492] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12: Bootstrap starting.
I20260808 20:12:53.877136 11347 heartbeater.cc:503] Master 127.10.24.62:46075 was elected leader, sending a full tablet report...
I20260808 20:12:53.878000 11593 ts_tablet_manager.cc:1428] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260808 20:12:53.878726 11592 tablet_bootstrap.cc:654] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:53.878811 11593 tablet_bootstrap.cc:492] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8: Bootstrap starting.
I20260808 20:12:53.888780 11594 tablet_bootstrap.cc:492] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98: No bootstrap required, opened a new log
I20260808 20:12:53.889245 11594 ts_tablet_manager.cc:1397] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.005s
I20260808 20:12:53.889966 11592 tablet_bootstrap.cc:492] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12: No bootstrap required, opened a new log
I20260808 20:12:53.890396 11592 ts_tablet_manager.cc:1397] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.014s
I20260808 20:12:53.890923 11593 tablet_bootstrap.cc:654] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:53.891788 11594 raft_consensus.cc:348] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:53.892493 11594 raft_consensus.cc:374] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:53.892773 11594 raft_consensus.cc:729] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa933dd5405492aa709133798d4ba98, State: Initialized, Role: FOLLOWER
I20260808 20:12:53.892748 11592 raft_consensus.cc:348] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:53.893853 11592 raft_consensus.cc:374] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:53.893590 11594 consensus_queue.cc:260] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:53.894249 11592 raft_consensus.cc:729] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Initialized, Role: FOLLOWER
I20260808 20:12:53.896631 11592 consensus_queue.cc:260] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:53.899070 11592 ts_tablet_manager.cc:1428] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260808 20:12:53.899905 11601 raft_consensus.cc:482] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:53.900369 11601 raft_consensus.cc:504] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:53.905642 11601 leader_election.cc:288] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075)
I20260808 20:12:53.907828 11592 tablet_bootstrap.cc:492] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12: Bootstrap starting.
I20260808 20:12:53.913151 11592 tablet_bootstrap.cc:654] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:53.915241 11455 heartbeater.cc:503] Master 127.10.24.62:46075 was elected leader, sending a full tablet report...
I20260808 20:12:53.920603 11593 tablet_bootstrap.cc:492] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8: No bootstrap required, opened a new log
I20260808 20:12:53.921087 11593 ts_tablet_manager.cc:1397] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8: Time spent bootstrapping tablet: real 0.042s user 0.024s sys 0.005s
I20260808 20:12:53.923522 11593 raft_consensus.cc:348] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:53.924222 11593 raft_consensus.cc:374] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:53.924516 11593 raft_consensus.cc:729] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Initialized, Role: FOLLOWER
I20260808 20:12:53.930343 11418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44506b0c529c45efbee5ac3003ef6f42" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa933dd5405492aa709133798d4ba98" is_pre_election: true
I20260808 20:12:53.931166 11418 raft_consensus.cc:2454] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 0.
I20260808 20:12:53.931736 11593 consensus_queue.cc:260] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:53.932678 11475 leader_election.cc:302] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, 2fa933dd5405492aa709133798d4ba98; no voters:
I20260808 20:12:53.933480 11601 raft_consensus.cc:2781] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:12:53.933797 11601 raft_consensus.cc:482] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:12:53.934166 11601 raft_consensus.cc:3037] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:53.936630 11594 ts_tablet_manager.cc:1428] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98: Time spent starting tablet: real 0.047s user 0.005s sys 0.007s
I20260808 20:12:53.937659 11594 tablet_bootstrap.cc:492] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98: Bootstrap starting.
I20260808 20:12:53.940670 11601 raft_consensus.cc:504] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:53.942374 11601 leader_election.cc:288] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 election: Requested vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075)
I20260808 20:12:53.943498 11594 tablet_bootstrap.cc:654] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:53.943332 11418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44506b0c529c45efbee5ac3003ef6f42" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa933dd5405492aa709133798d4ba98"
I20260808 20:12:53.943981 11418 raft_consensus.cc:3037] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:53.947517 11310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44506b0c529c45efbee5ac3003ef6f42" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
I20260808 20:12:53.947602 11311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44506b0c529c45efbee5ac3003ef6f42" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" is_pre_election: true
W20260808 20:12:53.949046 11472 leader_election.cc:341] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:12:53.933851 11593 ts_tablet_manager.cc:1428] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
W20260808 20:12:53.949760 11472 leader_election.cc:341] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:12:53.950225 11418 raft_consensus.cc:2454] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 1.
I20260808 20:12:53.951232 11475 leader_election.cc:302] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, 2fa933dd5405492aa709133798d4ba98; no voters: bb8edc2375f449ac806aebab2d5fd6a8
I20260808 20:12:53.952363 11592 tablet_bootstrap.cc:492] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12: No bootstrap required, opened a new log
I20260808 20:12:53.952721 11593 tablet_bootstrap.cc:492] T 07fe59ac00b24efb8271f8e86a33441b P bb8edc2375f449ac806aebab2d5fd6a8: Bootstrap starting.
I20260808 20:12:53.952737 11601 raft_consensus.cc:2781] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:12:53.953939 11592 ts_tablet_manager.cc:1397] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12: Time spent bootstrapping tablet: real 0.045s user 0.010s sys 0.017s
I20260808 20:12:53.956226 11592 raft_consensus.cc:348] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:53.956884 11592 raft_consensus.cc:374] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:53.957125 11592 raft_consensus.cc:729] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Initialized, Role: FOLLOWER
I20260808 20:12:53.957660 11592 consensus_queue.cc:260] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:53.958892 11593 tablet_bootstrap.cc:654] T 07fe59ac00b24efb8271f8e86a33441b P bb8edc2375f449ac806aebab2d5fd6a8: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:53.961690 11592 ts_tablet_manager.cc:1428] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260808 20:12:53.962422 11592 tablet_bootstrap.cc:492] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12: Bootstrap starting.
I20260808 20:12:53.964192 11594 tablet_bootstrap.cc:492] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98: No bootstrap required, opened a new log
I20260808 20:12:53.964625 11594 ts_tablet_manager.cc:1397] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.009s
I20260808 20:12:53.966586 11601 raft_consensus.cc:686] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Becoming Leader. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Running, Role: LEADER
I20260808 20:12:53.966914 11593 tablet_bootstrap.cc:492] T 07fe59ac00b24efb8271f8e86a33441b P bb8edc2375f449ac806aebab2d5fd6a8: No bootstrap required, opened a new log
I20260808 20:12:53.967355 11593 ts_tablet_manager.cc:1397] T 07fe59ac00b24efb8271f8e86a33441b P bb8edc2375f449ac806aebab2d5fd6a8: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20260808 20:12:53.967222 11601 consensus_queue.cc:237] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:53.967235 11594 raft_consensus.cc:348] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:53.968153 11594 raft_consensus.cc:374] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:53.968434 11594 raft_consensus.cc:729] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa933dd5405492aa709133798d4ba98, State: Initialized, Role: FOLLOWER
I20260808 20:12:53.969123 11594 consensus_queue.cc:260] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:53.969936 11593 raft_consensus.cc:348] T 07fe59ac00b24efb8271f8e86a33441b P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:53.970613 11593 raft_consensus.cc:374] T 07fe59ac00b24efb8271f8e86a33441b P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:53.970882 11593 raft_consensus.cc:729] T 07fe59ac00b24efb8271f8e86a33441b P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Initialized, Role: FOLLOWER
I20260808 20:12:53.971517 11593 consensus_queue.cc:260] T 07fe59ac00b24efb8271f8e86a33441b P bb8edc2375f449ac806aebab2d5fd6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:53.972048 11592 tablet_bootstrap.cc:654] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:53.973575 11593 ts_tablet_manager.cc:1428] T 07fe59ac00b24efb8271f8e86a33441b P bb8edc2375f449ac806aebab2d5fd6a8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 20:12:53.977278 11594 ts_tablet_manager.cc:1428] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260808 20:12:53.981755 11593 tablet_bootstrap.cc:492] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8: Bootstrap starting.
I20260808 20:12:53.983520 11592 tablet_bootstrap.cc:492] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12: No bootstrap required, opened a new log
I20260808 20:12:53.984004 11592 ts_tablet_manager.cc:1397] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.007s
I20260808 20:12:53.985421 11163 catalog_manager.cc:5495] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 124d7af2a136423c9165edf5e240ec12 (127.10.24.3). New cstate: current_term: 1 leader_uuid: "124d7af2a136423c9165edf5e240ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: HEALTHY } } }
I20260808 20:12:53.986568 11592 raft_consensus.cc:348] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:53.987360 11592 raft_consensus.cc:374] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:53.987571 11592 raft_consensus.cc:729] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Initialized, Role: FOLLOWER
I20260808 20:12:53.987645 11593 tablet_bootstrap.cc:654] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:53.988168 11592 consensus_queue.cc:260] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:53.989315 11594 tablet_bootstrap.cc:492] T 07fe59ac00b24efb8271f8e86a33441b P 2fa933dd5405492aa709133798d4ba98: Bootstrap starting.
I20260808 20:12:53.990033 11592 ts_tablet_manager.cc:1428] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260808 20:12:53.990863 11592 tablet_bootstrap.cc:492] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12: Bootstrap starting.
I20260808 20:12:53.995359 11592 tablet_bootstrap.cc:654] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:53.998016 11594 tablet_bootstrap.cc:654] T 07fe59ac00b24efb8271f8e86a33441b P 2fa933dd5405492aa709133798d4ba98: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:54.005899 11593 tablet_bootstrap.cc:492] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8: No bootstrap required, opened a new log
I20260808 20:12:54.005983 11594 tablet_bootstrap.cc:492] T 07fe59ac00b24efb8271f8e86a33441b P 2fa933dd5405492aa709133798d4ba98: No bootstrap required, opened a new log
I20260808 20:12:54.006274 11593 ts_tablet_manager.cc:1397] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.012s
I20260808 20:12:54.006431 11592 tablet_bootstrap.cc:492] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12: No bootstrap required, opened a new log
I20260808 20:12:54.006422 11594 ts_tablet_manager.cc:1397] T 07fe59ac00b24efb8271f8e86a33441b P 2fa933dd5405492aa709133798d4ba98: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.001s
I20260808 20:12:54.006958 11592 ts_tablet_manager.cc:1397] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.013s
I20260808 20:12:54.008152 11593 raft_consensus.cc:348] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.008725 11593 raft_consensus.cc:374] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:54.008963 11593 raft_consensus.cc:729] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Initialized, Role: FOLLOWER
I20260808 20:12:54.009420 11601 raft_consensus.cc:482] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:54.009501 11592 raft_consensus.cc:348] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.009378 11594 raft_consensus.cc:348] T 07fe59ac00b24efb8271f8e86a33441b P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.009562 11593 consensus_queue.cc:260] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.010345 11592 raft_consensus.cc:374] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:54.010377 11594 raft_consensus.cc:374] T 07fe59ac00b24efb8271f8e86a33441b P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:54.010191 11601 raft_consensus.cc:504] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.010767 11592 raft_consensus.cc:729] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Initialized, Role: FOLLOWER
I20260808 20:12:54.010802 11594 raft_consensus.cc:729] T 07fe59ac00b24efb8271f8e86a33441b P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa933dd5405492aa709133798d4ba98, State: Initialized, Role: FOLLOWER
I20260808 20:12:54.012071 11601 leader_election.cc:288] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075)
I20260808 20:12:54.011642 11594 consensus_queue.cc:260] T 07fe59ac00b24efb8271f8e86a33441b P 2fa933dd5405492aa709133798d4ba98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.012773 11311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "261b728800574016968c1083c414e27f" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" is_pre_election: true
I20260808 20:12:54.013363 11311 raft_consensus.cc:2454] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 0.
I20260808 20:12:54.013458 11418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "261b728800574016968c1083c414e27f" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa933dd5405492aa709133798d4ba98" is_pre_election: true
I20260808 20:12:54.014320 11594 ts_tablet_manager.cc:1428] T 07fe59ac00b24efb8271f8e86a33441b P 2fa933dd5405492aa709133798d4ba98: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260808 20:12:54.014580 11472 leader_election.cc:302] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, bb8edc2375f449ac806aebab2d5fd6a8; no voters:
I20260808 20:12:54.015201 11594 tablet_bootstrap.cc:492] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98: Bootstrap starting.
I20260808 20:12:54.015355 11601 raft_consensus.cc:2781] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:12:54.011613 11592 consensus_queue.cc:260] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.015664 11601 raft_consensus.cc:482] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:12:54.017575 11601 raft_consensus.cc:3037] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.017786 11592 ts_tablet_manager.cc:1428] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20260808 20:12:54.018630 11592 tablet_bootstrap.cc:492] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12: Bootstrap starting.
W20260808 20:12:54.015658 11475 leader_election.cc:341] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:12:54.024197 11592 tablet_bootstrap.cc:654] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:54.024533 11601 raft_consensus.cc:504] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.026852 11594 tablet_bootstrap.cc:654] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:54.027032 11311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "261b728800574016968c1083c414e27f" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
I20260808 20:12:54.027593 11311 raft_consensus.cc:3037] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.032508 11311 raft_consensus.cc:2454] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 1.
I20260808 20:12:54.033381 11472 leader_election.cc:302] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, bb8edc2375f449ac806aebab2d5fd6a8; no voters:
I20260808 20:12:54.034291 11611 raft_consensus.cc:2781] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:12:54.034761 11611 raft_consensus.cc:686] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Becoming Leader. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Running, Role: LEADER
I20260808 20:12:54.035578 11611 consensus_queue.cc:237] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.036922 11418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "261b728800574016968c1083c414e27f" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa933dd5405492aa709133798d4ba98"
W20260808 20:12:54.038303 11475 leader_election.cc:341] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:12:54.034752 11601 leader_election.cc:288] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 election: Requested vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075)
I20260808 20:12:54.034286 11593 ts_tablet_manager.cc:1428] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8: Time spent starting tablet: real 0.028s user 0.003s sys 0.004s
I20260808 20:12:54.041916 11593 tablet_bootstrap.cc:492] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8: Bootstrap starting.
I20260808 20:12:54.047362 11594 tablet_bootstrap.cc:492] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98: No bootstrap required, opened a new log
I20260808 20:12:54.047818 11594 ts_tablet_manager.cc:1397] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.004s
I20260808 20:12:54.047969 11593 tablet_bootstrap.cc:654] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:54.050067 11592 tablet_bootstrap.cc:492] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12: No bootstrap required, opened a new log
I20260808 20:12:54.050621 11592 ts_tablet_manager.cc:1397] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12: Time spent bootstrapping tablet: real 0.032s user 0.021s sys 0.007s
I20260808 20:12:54.050534 11594 raft_consensus.cc:348] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } } peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } }
I20260808 20:12:54.051448 11594 raft_consensus.cc:374] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:54.051759 11594 raft_consensus.cc:729] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa933dd5405492aa709133798d4ba98, State: Initialized, Role: FOLLOWER
I20260808 20:12:54.052583 11594 consensus_queue.cc:260] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } } peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } }
I20260808 20:12:54.053548 11592 raft_consensus.cc:348] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.054302 11592 raft_consensus.cc:374] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:54.054612 11592 raft_consensus.cc:729] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Initialized, Role: FOLLOWER
I20260808 20:12:54.055389 11592 consensus_queue.cc:260] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.056401 11163 catalog_manager.cc:5495] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 124d7af2a136423c9165edf5e240ec12 (127.10.24.3). New cstate: current_term: 1 leader_uuid: "124d7af2a136423c9165edf5e240ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: HEALTHY } } }
I20260808 20:12:54.057642 11592 ts_tablet_manager.cc:1428] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260808 20:12:54.058532 11592 tablet_bootstrap.cc:492] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12: Bootstrap starting.
I20260808 20:12:54.061620 11594 ts_tablet_manager.cc:1428] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98: Time spent starting tablet: real 0.013s user 0.004s sys 0.004s
I20260808 20:12:54.064111 11592 tablet_bootstrap.cc:654] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:54.069387 11594 tablet_bootstrap.cc:492] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98: Bootstrap starting.
I20260808 20:12:54.075799 11592 tablet_bootstrap.cc:492] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12: No bootstrap required, opened a new log
I20260808 20:12:54.076009 11594 tablet_bootstrap.cc:654] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:54.076193 11592 ts_tablet_manager.cc:1397] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.006s
I20260808 20:12:54.076241 11593 tablet_bootstrap.cc:492] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8: No bootstrap required, opened a new log
I20260808 20:12:54.076792 11593 ts_tablet_manager.cc:1397] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.028s
I20260808 20:12:54.078573 11592 raft_consensus.cc:348] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } } peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } }
I20260808 20:12:54.079087 11592 raft_consensus.cc:374] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:54.079325 11592 raft_consensus.cc:729] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Initialized, Role: FOLLOWER
I20260808 20:12:54.079577 11593 raft_consensus.cc:348] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.079883 11592 consensus_queue.cc:260] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } } peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } }
I20260808 20:12:54.080338 11593 raft_consensus.cc:374] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:54.080657 11593 raft_consensus.cc:729] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Initialized, Role: FOLLOWER
I20260808 20:12:54.081868 11592 ts_tablet_manager.cc:1428] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260808 20:12:54.081466 11593 consensus_queue.cc:260] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.082753 11592 tablet_bootstrap.cc:492] T f2719138be134d1382e8c595a0275d7f P 124d7af2a136423c9165edf5e240ec12: Bootstrap starting.
I20260808 20:12:54.088632 11592 tablet_bootstrap.cc:654] T f2719138be134d1382e8c595a0275d7f P 124d7af2a136423c9165edf5e240ec12: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:54.089479 11593 ts_tablet_manager.cc:1428] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20260808 20:12:54.096668 11593 tablet_bootstrap.cc:492] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8: Bootstrap starting.
I20260808 20:12:54.099115 11592 tablet_bootstrap.cc:492] T f2719138be134d1382e8c595a0275d7f P 124d7af2a136423c9165edf5e240ec12: No bootstrap required, opened a new log
I20260808 20:12:54.099488 11592 ts_tablet_manager.cc:1397] T f2719138be134d1382e8c595a0275d7f P 124d7af2a136423c9165edf5e240ec12: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.003s
I20260808 20:12:54.101822 11594 tablet_bootstrap.cc:492] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98: No bootstrap required, opened a new log
I20260808 20:12:54.101608 11592 raft_consensus.cc:348] T f2719138be134d1382e8c595a0275d7f P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.102021 11593 tablet_bootstrap.cc:654] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:54.102161 11592 raft_consensus.cc:374] T f2719138be134d1382e8c595a0275d7f P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:54.102273 11594 ts_tablet_manager.cc:1397] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98: Time spent bootstrapping tablet: real 0.033s user 0.012s sys 0.017s
I20260808 20:12:54.102469 11592 raft_consensus.cc:729] T f2719138be134d1382e8c595a0275d7f P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Initialized, Role: FOLLOWER
I20260808 20:12:54.103080 11592 consensus_queue.cc:260] T f2719138be134d1382e8c595a0275d7f P 124d7af2a136423c9165edf5e240ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.104722 11592 ts_tablet_manager.cc:1428] T f2719138be134d1382e8c595a0275d7f P 124d7af2a136423c9165edf5e240ec12: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260808 20:12:54.104863 11594 raft_consensus.cc:348] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.105643 11592 tablet_bootstrap.cc:492] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12: Bootstrap starting.
I20260808 20:12:54.105643 11594 raft_consensus.cc:374] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:54.106113 11594 raft_consensus.cc:729] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa933dd5405492aa709133798d4ba98, State: Initialized, Role: FOLLOWER
I20260808 20:12:54.106779 11594 consensus_queue.cc:260] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.111171 11592 tablet_bootstrap.cc:654] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:54.114882 11594 ts_tablet_manager.cc:1428] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98: Time spent starting tablet: real 0.012s user 0.003s sys 0.004s
I20260808 20:12:54.115886 11594 tablet_bootstrap.cc:492] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98: Bootstrap starting.
I20260808 20:12:54.122031 11592 tablet_bootstrap.cc:492] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12: No bootstrap required, opened a new log
I20260808 20:12:54.122459 11592 ts_tablet_manager.cc:1397] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20260808 20:12:54.122797 11594 tablet_bootstrap.cc:654] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:54.124732 11592 raft_consensus.cc:348] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.125396 11592 raft_consensus.cc:374] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:54.125598 11592 raft_consensus.cc:729] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Initialized, Role: FOLLOWER
I20260808 20:12:54.126174 11593 tablet_bootstrap.cc:492] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8: No bootstrap required, opened a new log
I20260808 20:12:54.126287 11592 consensus_queue.cc:260] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.126603 11593 ts_tablet_manager.cc:1397] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8: Time spent bootstrapping tablet: real 0.030s user 0.023s sys 0.004s
I20260808 20:12:54.129205 11593 raft_consensus.cc:348] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.129719 11592 ts_tablet_manager.cc:1428] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260808 20:12:54.129928 11593 raft_consensus.cc:374] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:54.130255 11593 raft_consensus.cc:729] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Initialized, Role: FOLLOWER
I20260808 20:12:54.130534 11592 tablet_bootstrap.cc:492] T f616636ffca943a7a1731ee069940307 P 124d7af2a136423c9165edf5e240ec12: Bootstrap starting.
I20260808 20:12:54.131047 11593 consensus_queue.cc:260] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.135612 11592 tablet_bootstrap.cc:654] T f616636ffca943a7a1731ee069940307 P 124d7af2a136423c9165edf5e240ec12: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:54.139401 11593 ts_tablet_manager.cc:1428] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260808 20:12:54.142740 11593 tablet_bootstrap.cc:492] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8: Bootstrap starting.
I20260808 20:12:54.146313 11592 tablet_bootstrap.cc:492] T f616636ffca943a7a1731ee069940307 P 124d7af2a136423c9165edf5e240ec12: No bootstrap required, opened a new log
I20260808 20:12:54.146694 11592 ts_tablet_manager.cc:1397] T f616636ffca943a7a1731ee069940307 P 124d7af2a136423c9165edf5e240ec12: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.008s
I20260808 20:12:54.148273 11592 raft_consensus.cc:348] T f616636ffca943a7a1731ee069940307 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.148711 11592 raft_consensus.cc:374] T f616636ffca943a7a1731ee069940307 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:54.148964 11592 raft_consensus.cc:729] T f616636ffca943a7a1731ee069940307 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Initialized, Role: FOLLOWER
I20260808 20:12:54.149096 11594 tablet_bootstrap.cc:492] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98: No bootstrap required, opened a new log
I20260808 20:12:54.149535 11594 ts_tablet_manager.cc:1397] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98: Time spent bootstrapping tablet: real 0.034s user 0.015s sys 0.014s
I20260808 20:12:54.149647 11593 tablet_bootstrap.cc:654] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:54.149475 11592 consensus_queue.cc:260] T f616636ffca943a7a1731ee069940307 P 124d7af2a136423c9165edf5e240ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.151193 11592 ts_tablet_manager.cc:1428] T f616636ffca943a7a1731ee069940307 P 124d7af2a136423c9165edf5e240ec12: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260808 20:12:54.152103 11594 raft_consensus.cc:348] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.152838 11594 raft_consensus.cc:374] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:54.153149 11594 raft_consensus.cc:729] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa933dd5405492aa709133798d4ba98, State: Initialized, Role: FOLLOWER
I20260808 20:12:54.153781 11594 consensus_queue.cc:260] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.155246 11593 tablet_bootstrap.cc:492] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8: No bootstrap required, opened a new log
I20260808 20:12:54.155710 11593 ts_tablet_manager.cc:1397] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260808 20:12:54.155829 11594 ts_tablet_manager.cc:1428] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260808 20:12:54.156690 11594 tablet_bootstrap.cc:492] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98: Bootstrap starting.
I20260808 20:12:54.158306 11593 raft_consensus.cc:348] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.159051 11593 raft_consensus.cc:374] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:54.159374 11593 raft_consensus.cc:729] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Initialized, Role: FOLLOWER
I20260808 20:12:54.160120 11593 consensus_queue.cc:260] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.162191 11594 tablet_bootstrap.cc:654] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:54.163406 11605 raft_consensus.cc:482] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:54.163875 11605 raft_consensus.cc:504] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } } peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } }
I20260808 20:12:54.164974 11611 raft_consensus.cc:482] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:54.165400 11611 raft_consensus.cc:504] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.167369 11611 leader_election.cc:288] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075)
I20260808 20:12:54.168740 11593 ts_tablet_manager.cc:1428] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260808 20:12:54.168793 11311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a5465239d74c75bfae9807d91bccba" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" is_pre_election: true
I20260808 20:12:54.169201 11418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a5465239d74c75bfae9807d91bccba" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa933dd5405492aa709133798d4ba98" is_pre_election: true
I20260808 20:12:54.169523 11311 raft_consensus.cc:2454] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 0.
I20260808 20:12:54.170152 11593 tablet_bootstrap.cc:492] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8: Bootstrap starting.
W20260808 20:12:54.170423 11475 leader_election.cc:341] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:12:54.170671 11600 raft_consensus.cc:482] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:54.170974 11611 raft_consensus.cc:482] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:54.171087 11600 raft_consensus.cc:504] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } } peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } }
I20260808 20:12:54.171387 11611 raft_consensus.cc:504] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.173422 11472 leader_election.cc:302] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, bb8edc2375f449ac806aebab2d5fd6a8; no voters: 2fa933dd5405492aa709133798d4ba98
I20260808 20:12:54.174065 11418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fe59ac00b24efb8271f8e86a33441b" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa933dd5405492aa709133798d4ba98" is_pre_election: true
I20260808 20:12:54.174440 11601 raft_consensus.cc:2781] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:12:54.174532 11594 tablet_bootstrap.cc:492] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98: No bootstrap required, opened a new log
I20260808 20:12:54.174647 11418 raft_consensus.cc:2454] T 07fe59ac00b24efb8271f8e86a33441b P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 0.
I20260808 20:12:54.174827 11601 raft_consensus.cc:482] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:12:54.175125 11594 ts_tablet_manager.cc:1397] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20260808 20:12:54.175730 11475 leader_election.cc:302] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, 2fa933dd5405492aa709133798d4ba98; no voters:
I20260808 20:12:54.175954 11311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fe59ac00b24efb8271f8e86a33441b" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" is_pre_election: true
I20260808 20:12:54.176506 11311 raft_consensus.cc:2454] T 07fe59ac00b24efb8271f8e86a33441b P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 0.
I20260808 20:12:54.176631 11612 raft_consensus.cc:2781] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:12:54.176990 11612 raft_consensus.cc:482] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:12:54.177203 11593 tablet_bootstrap.cc:654] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:54.177310 11612 raft_consensus.cc:3037] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.177657 11594 raft_consensus.cc:348] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.178027 11605 leader_election.cc:288] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783), bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959)
I20260808 20:12:54.183010 11594 raft_consensus.cc:374] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:54.183300 11594 raft_consensus.cc:729] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa933dd5405492aa709133798d4ba98, State: Initialized, Role: FOLLOWER
I20260808 20:12:54.183440 11611 leader_election.cc:288] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075)
I20260808 20:12:54.178332 11600 leader_election.cc:288] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783), 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075)
I20260808 20:12:54.175181 11601 raft_consensus.cc:3037] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.184852 11594 consensus_queue.cc:260] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.188427 11594 ts_tablet_manager.cc:1428] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260808 20:12:54.188488 11612 raft_consensus.cc:504] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.189406 11594 tablet_bootstrap.cc:492] T d6a5465239d74c75bfae9807d91bccba P 2fa933dd5405492aa709133798d4ba98: Bootstrap starting.
I20260808 20:12:54.194044 11601 raft_consensus.cc:504] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.192375 11612 leader_election.cc:288] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 election: Requested vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075)
I20260808 20:12:54.195850 11601 leader_election.cc:288] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 election: Requested vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075)
I20260808 20:12:54.200134 11594 tablet_bootstrap.cc:654] T d6a5465239d74c75bfae9807d91bccba P 2fa933dd5405492aa709133798d4ba98: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:54.202672 11418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fe59ac00b24efb8271f8e86a33441b" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa933dd5405492aa709133798d4ba98"
I20260808 20:12:54.203262 11418 raft_consensus.cc:3037] T 07fe59ac00b24efb8271f8e86a33441b P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.203420 11593 tablet_bootstrap.cc:492] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8: No bootstrap required, opened a new log
I20260808 20:12:54.204008 11593 ts_tablet_manager.cc:1397] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.005s
I20260808 20:12:54.205255 11310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a5465239d74c75bfae9807d91bccba" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
I20260808 20:12:54.203892 11311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fe59ac00b24efb8271f8e86a33441b" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
I20260808 20:12:54.205854 11310 raft_consensus.cc:3037] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.205951 11311 raft_consensus.cc:3037] T 07fe59ac00b24efb8271f8e86a33441b P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.209524 11418 raft_consensus.cc:2454] T 07fe59ac00b24efb8271f8e86a33441b P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 1.
I20260808 20:12:54.210100 11593 raft_consensus.cc:348] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.211014 11475 leader_election.cc:302] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, 2fa933dd5405492aa709133798d4ba98; no voters:
I20260808 20:12:54.213018 11601 raft_consensus.cc:2781] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:12:54.213513 11601 raft_consensus.cc:686] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Becoming Leader. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Running, Role: LEADER
W20260808 20:12:54.214119 11226 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 20:12:54.212275 11310 raft_consensus.cc:2454] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 1.
I20260808 20:12:54.214285 11601 consensus_queue.cc:237] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.216045 11472 leader_election.cc:302] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, bb8edc2375f449ac806aebab2d5fd6a8; no voters:
I20260808 20:12:54.216758 11612 raft_consensus.cc:2781] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:12:54.217207 11612 raft_consensus.cc:686] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Becoming Leader. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Running, Role: LEADER
I20260808 20:12:54.218166 11593 raft_consensus.cc:374] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:54.217957 11612 consensus_queue.cc:237] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.211730 11418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a5465239d74c75bfae9807d91bccba" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa933dd5405492aa709133798d4ba98"
I20260808 20:12:54.218501 11593 raft_consensus.cc:729] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Initialized, Role: FOLLOWER
W20260808 20:12:54.219482 11475 leader_election.cc:341] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:12:54.219916 11593 consensus_queue.cc:260] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.227527 11593 ts_tablet_manager.cc:1428] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8: Time spent starting tablet: real 0.023s user 0.004s sys 0.001s
I20260808 20:12:54.230320 11163 catalog_manager.cc:5495] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 124d7af2a136423c9165edf5e240ec12 (127.10.24.3). New cstate: current_term: 1 leader_uuid: "124d7af2a136423c9165edf5e240ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: HEALTHY } } }
I20260808 20:12:54.231346 11594 tablet_bootstrap.cc:492] T d6a5465239d74c75bfae9807d91bccba P 2fa933dd5405492aa709133798d4ba98: No bootstrap required, opened a new log
I20260808 20:12:54.231817 11594 ts_tablet_manager.cc:1397] T d6a5465239d74c75bfae9807d91bccba P 2fa933dd5405492aa709133798d4ba98: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.007s
I20260808 20:12:54.234333 11594 raft_consensus.cc:348] T d6a5465239d74c75bfae9807d91bccba P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.234999 11594 raft_consensus.cc:374] T d6a5465239d74c75bfae9807d91bccba P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:54.235257 11594 raft_consensus.cc:729] T d6a5465239d74c75bfae9807d91bccba P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa933dd5405492aa709133798d4ba98, State: Initialized, Role: FOLLOWER
I20260808 20:12:54.235893 11594 consensus_queue.cc:260] T d6a5465239d74c75bfae9807d91bccba P 2fa933dd5405492aa709133798d4ba98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.236364 11532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4e13f27b9d4abbbda7599371e4ad48" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124d7af2a136423c9165edf5e240ec12" is_pre_election: true
I20260808 20:12:54.222375 11311 raft_consensus.cc:2454] T 07fe59ac00b24efb8271f8e86a33441b P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 1.
I20260808 20:12:54.238168 11532 raft_consensus.cc:2454] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fa933dd5405492aa709133798d4ba98 in term 0.
I20260808 20:12:54.239470 11370 leader_election.cc:302] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, 2fa933dd5405492aa709133798d4ba98; no voters:
I20260808 20:12:54.240283 11605 raft_consensus.cc:2781] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:12:54.240612 11605 raft_consensus.cc:482] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:12:54.240970 11605 raft_consensus.cc:3037] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.244602 11594 ts_tablet_manager.cc:1428] T d6a5465239d74c75bfae9807d91bccba P 2fa933dd5405492aa709133798d4ba98: Time spent starting tablet: real 0.013s user 0.001s sys 0.005s
I20260808 20:12:54.248209 11310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4e13f27b9d4abbbda7599371e4ad48" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" is_pre_election: true
I20260808 20:12:54.248123 11605 raft_consensus.cc:504] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } } peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } }
I20260808 20:12:54.248911 11310 raft_consensus.cc:2454] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fa933dd5405492aa709133798d4ba98 in term 0.
I20260808 20:12:54.249963 11605 leader_election.cc:288] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 1 election: Requested vote from peers 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783), bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959)
I20260808 20:12:54.251215 11310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4e13f27b9d4abbbda7599371e4ad48" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
I20260808 20:12:54.251740 11310 raft_consensus.cc:3037] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.251665 11532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4e13f27b9d4abbbda7599371e4ad48" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124d7af2a136423c9165edf5e240ec12"
I20260808 20:12:54.252285 11532 raft_consensus.cc:3037] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.253402 11160 catalog_manager.cc:5495] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 124d7af2a136423c9165edf5e240ec12 (127.10.24.3). New cstate: current_term: 1 leader_uuid: "124d7af2a136423c9165edf5e240ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: HEALTHY } } }
I20260808 20:12:54.252478 11593 tablet_bootstrap.cc:492] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8: Bootstrap starting.
I20260808 20:12:54.258077 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4e13f27b9d4abbbda7599371e4ad48" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124d7af2a136423c9165edf5e240ec12" is_pre_election: true
I20260808 20:12:54.259634 11605 raft_consensus.cc:482] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:54.260061 11605 raft_consensus.cc:504] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.261183 11418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4e13f27b9d4abbbda7599371e4ad48" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa933dd5405492aa709133798d4ba98" is_pre_election: true
I20260808 20:12:54.262341 11605 leader_election.cc:288] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783)
I20260808 20:12:54.263059 11418 raft_consensus.cc:2379] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bb8edc2375f449ac806aebab2d5fd6a8 in current term 1: Already voted for candidate 2fa933dd5405492aa709133798d4ba98 in this term.
I20260808 20:12:54.262918 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f616636ffca943a7a1731ee069940307" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124d7af2a136423c9165edf5e240ec12" is_pre_election: true
I20260808 20:12:54.263684 11531 raft_consensus.cc:2454] T f616636ffca943a7a1731ee069940307 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fa933dd5405492aa709133798d4ba98 in term 0.
I20260808 20:12:54.264417 11257 leader_election.cc:302] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb8edc2375f449ac806aebab2d5fd6a8; no voters: 124d7af2a136423c9165edf5e240ec12, 2fa933dd5405492aa709133798d4ba98
I20260808 20:12:54.264771 11370 leader_election.cc:302] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, 2fa933dd5405492aa709133798d4ba98; no voters:
I20260808 20:12:54.265462 11593 tablet_bootstrap.cc:654] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:54.265908 11605 raft_consensus.cc:2781] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:12:54.263123 11532 raft_consensus.cc:2454] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fa933dd5405492aa709133798d4ba98 in term 1.
I20260808 20:12:54.265594 11311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f616636ffca943a7a1731ee069940307" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" is_pre_election: true
I20260808 20:12:54.266240 11605 raft_consensus.cc:482] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:12:54.267277 11370 leader_election.cc:302] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, 2fa933dd5405492aa709133798d4ba98; no voters:
I20260808 20:12:54.267684 11605 raft_consensus.cc:3037] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.267534 11311 raft_consensus.cc:2454] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fa933dd5405492aa709133798d4ba98 in term 0.
I20260808 20:12:54.268921 11310 raft_consensus.cc:2454] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fa933dd5405492aa709133798d4ba98 in term 1.
I20260808 20:12:54.269384 11600 raft_consensus.cc:2726] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:12:54.270054 11594 tablet_bootstrap.cc:492] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98: Bootstrap starting.
I20260808 20:12:54.273591 11612 raft_consensus.cc:482] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:54.274008 11612 raft_consensus.cc:504] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.274261 11605 raft_consensus.cc:504] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.274817 11594 tablet_bootstrap.cc:654] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:54.275866 11612 leader_election.cc:288] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075)
I20260808 20:12:54.276103 11605 leader_election.cc:288] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 1 election: Requested vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783)
I20260808 20:12:54.276629 11605 raft_consensus.cc:2781] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:12:54.277055 11605 raft_consensus.cc:686] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 1 LEADER]: Becoming Leader. State: Replica: 2fa933dd5405492aa709133798d4ba98, State: Running, Role: LEADER
I20260808 20:12:54.277834 11605 consensus_queue.cc:237] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } } peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } }
I20260808 20:12:54.277762 11310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304a2461fac34933aa3a07f54b875889" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" is_pre_election: true
I20260808 20:12:54.279199 11310 raft_consensus.cc:2454] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 0.
I20260808 20:12:54.279953 11532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f616636ffca943a7a1731ee069940307" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124d7af2a136423c9165edf5e240ec12"
I20260808 20:12:54.280236 11472 leader_election.cc:302] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, bb8edc2375f449ac806aebab2d5fd6a8; no voters:
I20260808 20:12:54.280488 11532 raft_consensus.cc:3037] T f616636ffca943a7a1731ee069940307 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.281198 11612 raft_consensus.cc:2781] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:12:54.281546 11612 raft_consensus.cc:482] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:12:54.281848 11612 raft_consensus.cc:3037] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.286525 11418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304a2461fac34933aa3a07f54b875889" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa933dd5405492aa709133798d4ba98" is_pre_election: true
W20260808 20:12:54.291378 11475 leader_election.cc:341] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:12:54.286964 11532 raft_consensus.cc:2454] T f616636ffca943a7a1731ee069940307 P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fa933dd5405492aa709133798d4ba98 in term 1.
I20260808 20:12:54.292625 11593 tablet_bootstrap.cc:492] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8: No bootstrap required, opened a new log
I20260808 20:12:54.293111 11593 ts_tablet_manager.cc:1397] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8: Time spent bootstrapping tablet: real 0.063s user 0.010s sys 0.006s
I20260808 20:12:54.294034 11310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f616636ffca943a7a1731ee069940307" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
I20260808 20:12:54.294716 11310 raft_consensus.cc:3037] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.295468 11593 raft_consensus.cc:348] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.296139 11593 raft_consensus.cc:374] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:54.296398 11593 raft_consensus.cc:729] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Initialized, Role: FOLLOWER
I20260808 20:12:54.289487 11656 raft_consensus.cc:482] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:54.297102 11656 raft_consensus.cc:504] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.299147 11656 leader_election.cc:288] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783)
I20260808 20:12:54.289934 11612 raft_consensus.cc:504] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.300769 11612 leader_election.cc:288] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 election: Requested vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075)
I20260808 20:12:54.302397 11418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304a2461fac34933aa3a07f54b875889" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa933dd5405492aa709133798d4ba98"
W20260808 20:12:54.303411 11475 leader_election.cc:341] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:12:54.302273 11593 consensus_queue.cc:260] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.303759 11309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "261b728800574016968c1083c414e27f" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" is_pre_election: true
I20260808 20:12:54.304498 11309 raft_consensus.cc:2379] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2fa933dd5405492aa709133798d4ba98 in current term 1: Already voted for candidate 124d7af2a136423c9165edf5e240ec12 in this term.
I20260808 20:12:54.305155 11593 ts_tablet_manager.cc:1428] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20260808 20:12:54.306214 11593 tablet_bootstrap.cc:492] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8: Bootstrap starting.
I20260808 20:12:54.307560 11594 tablet_bootstrap.cc:492] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98: No bootstrap required, opened a new log
I20260808 20:12:54.308024 11594 ts_tablet_manager.cc:1397] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98: Time spent bootstrapping tablet: real 0.038s user 0.021s sys 0.007s
I20260808 20:12:54.310479 11594 raft_consensus.cc:348] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.311491 11593 tablet_bootstrap.cc:654] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:54.312074 11594 raft_consensus.cc:374] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:54.312647 11594 raft_consensus.cc:729] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa933dd5405492aa709133798d4ba98, State: Initialized, Role: FOLLOWER
I20260808 20:12:54.313752 11594 consensus_queue.cc:260] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.316838 11594 ts_tablet_manager.cc:1428] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98: Time spent starting tablet: real 0.008s user 0.002s sys 0.003s
I20260808 20:12:54.320999 11311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304a2461fac34933aa3a07f54b875889" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
I20260808 20:12:54.325253 11311 raft_consensus.cc:3037] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.325699 11594 tablet_bootstrap.cc:492] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98: Bootstrap starting.
I20260808 20:12:54.323148 11160 catalog_manager.cc:5495] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fa933dd5405492aa709133798d4ba98 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "2fa933dd5405492aa709133798d4ba98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: HEALTHY } } }
I20260808 20:12:54.323825 11310 raft_consensus.cc:2454] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fa933dd5405492aa709133798d4ba98 in term 1.
I20260808 20:12:54.326227 11532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "261b728800574016968c1083c414e27f" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124d7af2a136423c9165edf5e240ec12" is_pre_election: true
I20260808 20:12:54.327811 11370 leader_election.cc:302] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, 2fa933dd5405492aa709133798d4ba98; no voters:
I20260808 20:12:54.328715 11659 raft_consensus.cc:2781] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:12:54.329413 11659 raft_consensus.cc:686] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [term 1 LEADER]: Becoming Leader. State: Replica: 2fa933dd5405492aa709133798d4ba98, State: Running, Role: LEADER
I20260808 20:12:54.330338 11659 consensus_queue.cc:237] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.331311 11594 tablet_bootstrap.cc:654] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98: Neither blocks nor log segments found. Creating new log.
I20260808 20:12:54.334510 11600 raft_consensus.cc:482] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:54.334879 11370 leader_election.cc:302] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fa933dd5405492aa709133798d4ba98; no voters: 124d7af2a136423c9165edf5e240ec12, bb8edc2375f449ac806aebab2d5fd6a8
I20260808 20:12:54.334952 11600 raft_consensus.cc:504] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.335714 11605 raft_consensus.cc:3037] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.336609 11612 raft_consensus.cc:482] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:54.336827 11600 leader_election.cc:288] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075), 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783)
I20260808 20:12:54.337530 11612 raft_consensus.cc:504] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.338742 11418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089736c28b834bd5a121f818bcb364d7" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa933dd5405492aa709133798d4ba98" is_pre_election: true
I20260808 20:12:54.339308 11418 raft_consensus.cc:2454] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb8edc2375f449ac806aebab2d5fd6a8 in term 0.
I20260808 20:12:54.339435 11612 leader_election.cc:288] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075)
I20260808 20:12:54.341210 11593 tablet_bootstrap.cc:492] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8: No bootstrap required, opened a new log
I20260808 20:12:54.341434 11417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089736c28b834bd5a121f818bcb364d7" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa933dd5405492aa709133798d4ba98" is_pre_election: true
I20260808 20:12:54.341708 11593 ts_tablet_manager.cc:1397] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8: Time spent bootstrapping tablet: real 0.036s user 0.014s sys 0.015s
I20260808 20:12:54.342054 11605 raft_consensus.cc:2726] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:12:54.342082 11417 raft_consensus.cc:2454] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 0.
I20260808 20:12:54.343277 11475 leader_election.cc:302] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, 2fa933dd5405492aa709133798d4ba98; no voters:
I20260808 20:12:54.343874 11612 raft_consensus.cc:2781] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:12:54.344121 11310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089736c28b834bd5a121f818bcb364d7" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" is_pre_election: true
I20260808 20:12:54.344213 11612 raft_consensus.cc:482] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:12:54.344715 11310 raft_consensus.cc:2454] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 0.
I20260808 20:12:54.344836 11612 raft_consensus.cc:3037] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.344353 11593 raft_consensus.cc:348] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.345793 11257 leader_election.cc:302] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fa933dd5405492aa709133798d4ba98, bb8edc2375f449ac806aebab2d5fd6a8; no voters:
I20260808 20:12:54.346221 11532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089736c28b834bd5a121f818bcb364d7" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124d7af2a136423c9165edf5e240ec12" is_pre_election: true
I20260808 20:12:54.345382 11593 raft_consensus.cc:374] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:54.346993 11593 raft_consensus.cc:729] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Initialized, Role: FOLLOWER
I20260808 20:12:54.347566 11593 consensus_queue.cc:260] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.348997 11600 raft_consensus.cc:2781] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:12:54.353636 11600 raft_consensus.cc:482] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:12:54.354058 11600 raft_consensus.cc:3037] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.355449 11605 raft_consensus.cc:482] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:54.355922 11605 raft_consensus.cc:504] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.357249 11594 tablet_bootstrap.cc:492] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98: No bootstrap required, opened a new log
I20260808 20:12:54.357733 11594 ts_tablet_manager.cc:1397] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98: Time spent bootstrapping tablet: real 0.032s user 0.014s sys 0.007s
I20260808 20:12:54.353531 11311 raft_consensus.cc:2454] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 1.
I20260808 20:12:54.358078 11605 leader_election.cc:288] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783)
I20260808 20:12:54.358927 11472 leader_election.cc:302] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, bb8edc2375f449ac806aebab2d5fd6a8; no voters: 2fa933dd5405492aa709133798d4ba98
I20260808 20:12:54.360003 11601 raft_consensus.cc:2781] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:12:54.360445 11601 raft_consensus.cc:686] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Becoming Leader. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Running, Role: LEADER
I20260808 20:12:54.361254 11601 consensus_queue.cc:237] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.364148 11160 catalog_manager.cc:5495] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fa933dd5405492aa709133798d4ba98 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "2fa933dd5405492aa709133798d4ba98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:54.352943 11612 raft_consensus.cc:504] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.365275 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304a2461fac34933aa3a07f54b875889" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124d7af2a136423c9165edf5e240ec12" is_pre_election: true
I20260808 20:12:54.358502 11605 raft_consensus.cc:482] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:54.368001 11310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089736c28b834bd5a121f818bcb364d7" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
I20260808 20:12:54.368957 11594 raft_consensus.cc:348] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.369583 11594 raft_consensus.cc:374] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:12:54.369812 11594 raft_consensus.cc:729] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa933dd5405492aa709133798d4ba98, State: Initialized, Role: FOLLOWER
I20260808 20:12:54.370440 11594 consensus_queue.cc:260] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.371526 11532 raft_consensus.cc:2379] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bb8edc2375f449ac806aebab2d5fd6a8 in current term 1: Already voted for candidate 124d7af2a136423c9165edf5e240ec12 in this term.
I20260808 20:12:54.371610 11311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304a2461fac34933aa3a07f54b875889" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" is_pre_election: true
I20260808 20:12:54.372232 11612 leader_election.cc:288] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 election: Requested vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075)
I20260808 20:12:54.372493 11594 ts_tablet_manager.cc:1428] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260808 20:12:54.373090 11311 raft_consensus.cc:2379] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2fa933dd5405492aa709133798d4ba98 in current term 1: Already voted for candidate 124d7af2a136423c9165edf5e240ec12 in this term.
I20260808 20:12:54.375054 11600 raft_consensus.cc:504] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.376137 11310 raft_consensus.cc:2379] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 124d7af2a136423c9165edf5e240ec12 in current term 1: Already voted for candidate bb8edc2375f449ac806aebab2d5fd6a8 in this term.
I20260808 20:12:54.376492 11370 leader_election.cc:302] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fa933dd5405492aa709133798d4ba98; no voters: 124d7af2a136423c9165edf5e240ec12, bb8edc2375f449ac806aebab2d5fd6a8
I20260808 20:12:54.377403 11418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089736c28b834bd5a121f818bcb364d7" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa933dd5405492aa709133798d4ba98"
I20260808 20:12:54.377945 11418 raft_consensus.cc:3037] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.379432 11417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089736c28b834bd5a121f818bcb364d7" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa933dd5405492aa709133798d4ba98"
I20260808 20:12:54.378396 11605 raft_consensus.cc:504] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.379109 11659 raft_consensus.cc:3037] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.381767 11600 leader_election.cc:288] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 1 election: Requested vote from peers 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075), 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783)
I20260808 20:12:54.384650 11532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089736c28b834bd5a121f818bcb364d7" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124d7af2a136423c9165edf5e240ec12"
I20260808 20:12:54.385358 11532 raft_consensus.cc:2379] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bb8edc2375f449ac806aebab2d5fd6a8 in current term 1: Already voted for candidate 124d7af2a136423c9165edf5e240ec12 in this term.
I20260808 20:12:54.386219 11257 leader_election.cc:302] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb8edc2375f449ac806aebab2d5fd6a8; no voters: 124d7af2a136423c9165edf5e240ec12, 2fa933dd5405492aa709133798d4ba98
I20260808 20:12:54.387424 11600 raft_consensus.cc:2726] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 20:12:54.388094 11310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2719138be134d1382e8c595a0275d7f" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" is_pre_election: true
I20260808 20:12:54.388689 11310 raft_consensus.cc:2454] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fa933dd5405492aa709133798d4ba98 in term 0.
I20260808 20:12:54.389961 11365 leader_election.cc:302] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fa933dd5405492aa709133798d4ba98, bb8edc2375f449ac806aebab2d5fd6a8; no voters:
I20260808 20:12:54.390458 11605 leader_election.cc:288] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783)
I20260808 20:12:54.390825 11656 raft_consensus.cc:2781] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:12:54.391178 11656 raft_consensus.cc:482] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:12:54.391227 11659 raft_consensus.cc:2726] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:12:54.391605 11656 raft_consensus.cc:3037] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.391956 11418 raft_consensus.cc:2454] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 1.
I20260808 20:12:54.393065 11593 ts_tablet_manager.cc:1428] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8: Time spent starting tablet: real 0.051s user 0.004s sys 0.008s
I20260808 20:12:54.397149 11532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2719138be134d1382e8c595a0275d7f" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124d7af2a136423c9165edf5e240ec12" is_pre_election: true
I20260808 20:12:54.397729 11532 raft_consensus.cc:2454] T f2719138be134d1382e8c595a0275d7f P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fa933dd5405492aa709133798d4ba98 in term 0.
I20260808 20:12:54.397817 11656 raft_consensus.cc:504] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.398851 11475 leader_election.cc:302] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, 2fa933dd5405492aa709133798d4ba98; no voters: bb8edc2375f449ac806aebab2d5fd6a8
I20260808 20:12:54.400204 11310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2719138be134d1382e8c595a0275d7f" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
I20260808 20:12:54.400599 11656 leader_election.cc:288] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 1 election: Requested vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783)
I20260808 20:12:54.400741 11310 raft_consensus.cc:3037] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.401798 11532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2719138be134d1382e8c595a0275d7f" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124d7af2a136423c9165edf5e240ec12"
I20260808 20:12:54.402418 11532 raft_consensus.cc:3037] T f2719138be134d1382e8c595a0275d7f P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.403947 11158 catalog_manager.cc:5495] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 124d7af2a136423c9165edf5e240ec12 (127.10.24.3). New cstate: current_term: 1 leader_uuid: "124d7af2a136423c9165edf5e240ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: HEALTHY } } }
I20260808 20:12:54.400498 11601 raft_consensus.cc:2781] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:12:54.405648 11601 raft_consensus.cc:686] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Becoming Leader. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Running, Role: LEADER
I20260808 20:12:54.407209 11310 raft_consensus.cc:2454] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fa933dd5405492aa709133798d4ba98 in term 1.
I20260808 20:12:54.408198 11365 leader_election.cc:302] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fa933dd5405492aa709133798d4ba98, bb8edc2375f449ac806aebab2d5fd6a8; no voters:
I20260808 20:12:54.406468 11601 consensus_queue.cc:237] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.409221 11656 raft_consensus.cc:2781] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:12:54.409722 11656 raft_consensus.cc:686] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [term 1 LEADER]: Becoming Leader. State: Replica: 2fa933dd5405492aa709133798d4ba98, State: Running, Role: LEADER
I20260808 20:12:54.411293 11656 consensus_queue.cc:237] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.416740 11532 raft_consensus.cc:2454] T f2719138be134d1382e8c595a0275d7f P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fa933dd5405492aa709133798d4ba98 in term 1.
I20260808 20:12:54.426328 11601 consensus_queue.cc:1035] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:54.433583 11601 consensus_queue.cc:1035] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:54.435328 11158 catalog_manager.cc:5495] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fa933dd5405492aa709133798d4ba98 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "2fa933dd5405492aa709133798d4ba98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:54.433048 11160 catalog_manager.cc:5495] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 124d7af2a136423c9165edf5e240ec12 (127.10.24.3). New cstate: current_term: 1 leader_uuid: "124d7af2a136423c9165edf5e240ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: HEALTHY } } }
I20260808 20:12:54.467332 11310 raft_consensus.cc:3037] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.478149 11612 consensus_queue.cc:1035] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260808 20:12:54.484320 11611 raft_consensus.cc:482] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:54.484845 11611 raft_consensus.cc:504] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.488252 11611 leader_election.cc:288] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075)
I20260808 20:12:54.488420 11311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941577432a4440b981675cc78e482325" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" is_pre_election: true
I20260808 20:12:54.489166 11311 raft_consensus.cc:2454] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 0.
I20260808 20:12:54.490294 11472 leader_election.cc:302] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, bb8edc2375f449ac806aebab2d5fd6a8; no voters:
I20260808 20:12:54.491039 11612 consensus_queue.cc:1035] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260808 20:12:54.490505 11417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941577432a4440b981675cc78e482325" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa933dd5405492aa709133798d4ba98" is_pre_election: true
I20260808 20:12:54.492220 11417 raft_consensus.cc:2454] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 0.
I20260808 20:12:54.496713 11612 raft_consensus.cc:2781] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:12:54.498054 11612 raft_consensus.cc:482] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:12:54.498400 11612 raft_consensus.cc:3037] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.506912 11612 raft_consensus.cc:504] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.509635 11310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941577432a4440b981675cc78e482325" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
I20260808 20:12:54.510241 11310 raft_consensus.cc:3037] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.516676 11310 raft_consensus.cc:2454] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 1.
I20260808 20:12:54.520601 11417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941577432a4440b981675cc78e482325" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa933dd5405492aa709133798d4ba98"
I20260808 20:12:54.521209 11417 raft_consensus.cc:3037] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.523533 11472 leader_election.cc:302] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, bb8edc2375f449ac806aebab2d5fd6a8; no voters:
I20260808 20:12:54.524761 11677 raft_consensus.cc:2781] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:12:54.525846 11677 raft_consensus.cc:686] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Becoming Leader. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Running, Role: LEADER
I20260808 20:12:54.526650 11677 consensus_queue.cc:237] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.534389 11417 raft_consensus.cc:2454] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 1.
I20260808 20:12:54.535059 11612 leader_election.cc:288] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 1 election: Requested vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075)
I20260808 20:12:54.540753 11160 catalog_manager.cc:5495] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 124d7af2a136423c9165edf5e240ec12 (127.10.24.3). New cstate: current_term: 1 leader_uuid: "124d7af2a136423c9165edf5e240ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: HEALTHY } } }
I20260808 20:12:54.569578 11417 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e4e13f27b9d4abbbda7599371e4ad48"
dest_uuid: "2fa933dd5405492aa709133798d4ba98"
mode: GRACEFUL
new_leader_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
from {username='slave'} at 127.0.0.1:47822
I20260808 20:12:54.570124 11417 raft_consensus.cc:595] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 1 LEADER]: Received request to transfer leadership to TS bb8edc2375f449ac806aebab2d5fd6a8
I20260808 20:12:54.582595 11532 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d6a5465239d74c75bfae9807d91bccba"
dest_uuid: "124d7af2a136423c9165edf5e240ec12"
mode: GRACEFUL
new_leader_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
from {username='slave'} at 127.0.0.1:52868
I20260808 20:12:54.583058 11532 raft_consensus.cc:595] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Received request to transfer leadership to TS bb8edc2375f449ac806aebab2d5fd6a8
I20260808 20:12:54.587080 11532 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "941577432a4440b981675cc78e482325"
dest_uuid: "124d7af2a136423c9165edf5e240ec12"
mode: GRACEFUL
new_leader_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
from {username='slave'} at 127.0.0.1:52868
I20260808 20:12:54.587536 11532 raft_consensus.cc:595] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Received request to transfer leadership to TS bb8edc2375f449ac806aebab2d5fd6a8
I20260808 20:12:54.591279 11532 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "261b728800574016968c1083c414e27f"
dest_uuid: "124d7af2a136423c9165edf5e240ec12"
mode: GRACEFUL
new_leader_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
from {username='slave'} at 127.0.0.1:52868
I20260808 20:12:54.591713 11532 raft_consensus.cc:595] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Received request to transfer leadership to TS bb8edc2375f449ac806aebab2d5fd6a8
I20260808 20:12:54.595496 11532 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44506b0c529c45efbee5ac3003ef6f42"
dest_uuid: "124d7af2a136423c9165edf5e240ec12"
mode: GRACEFUL
new_leader_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
from {username='slave'} at 127.0.0.1:52868
I20260808 20:12:54.595909 11532 raft_consensus.cc:595] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Received request to transfer leadership to TS bb8edc2375f449ac806aebab2d5fd6a8
I20260808 20:12:54.602092 11612 consensus_queue.cc:1035] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:54.604415 11532 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "304a2461fac34933aa3a07f54b875889"
dest_uuid: "124d7af2a136423c9165edf5e240ec12"
mode: GRACEFUL
new_leader_uuid: "2fa933dd5405492aa709133798d4ba98"
from {username='slave'} at 127.0.0.1:52868
I20260808 20:12:54.604846 11532 raft_consensus.cc:595] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Received request to transfer leadership to TS 2fa933dd5405492aa709133798d4ba98
I20260808 20:12:54.614275 11532 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "089736c28b834bd5a121f818bcb364d7"
dest_uuid: "124d7af2a136423c9165edf5e240ec12"
mode: GRACEFUL
new_leader_uuid: "2fa933dd5405492aa709133798d4ba98"
from {username='slave'} at 127.0.0.1:52868
I20260808 20:12:54.614502 11656 raft_consensus.cc:482] T d6a5465239d74c75bfae9807d91bccba P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:12:54.614849 11532 raft_consensus.cc:595] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Received request to transfer leadership to TS 2fa933dd5405492aa709133798d4ba98
I20260808 20:12:54.615018 11674 consensus_queue.cc:1035] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:12:54.614962 11656 raft_consensus.cc:504] T d6a5465239d74c75bfae9807d91bccba P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.616950 11656 leader_election.cc:288] T d6a5465239d74c75bfae9807d91bccba P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783)
I20260808 20:12:54.618811 11311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a5465239d74c75bfae9807d91bccba" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" is_pre_election: true
I20260808 20:12:54.619619 11311 raft_consensus.cc:2379] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2fa933dd5405492aa709133798d4ba98 in current term 1: Already voted for candidate 124d7af2a136423c9165edf5e240ec12 in this term.
I20260808 20:12:54.620430 11532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a5465239d74c75bfae9807d91bccba" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124d7af2a136423c9165edf5e240ec12" is_pre_election: true
I20260808 20:12:54.621599 11370 leader_election.cc:302] T d6a5465239d74c75bfae9807d91bccba P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fa933dd5405492aa709133798d4ba98; no voters: 124d7af2a136423c9165edf5e240ec12, bb8edc2375f449ac806aebab2d5fd6a8
I20260808 20:12:54.622378 11656 raft_consensus.cc:3037] T d6a5465239d74c75bfae9807d91bccba P 2fa933dd5405492aa709133798d4ba98 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:12:54.627041 11656 raft_consensus.cc:2726] T d6a5465239d74c75bfae9807d91bccba P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 20:12:54.633256 11348 tablet.cc:2369] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 20:12:54.678918 11656 consensus_queue.cc:1035] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:54.688167 11659 consensus_queue.cc:1035] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:54.694808 11659 raft_consensus.cc:979] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98: : Instructing follower bb8edc2375f449ac806aebab2d5fd6a8 to start an election
I20260808 20:12:54.695184 11605 raft_consensus.cc:1067] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 1 LEADER]: Signalling peer bb8edc2375f449ac806aebab2d5fd6a8 to start an election
I20260808 20:12:54.696470 11311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e4e13f27b9d4abbbda7599371e4ad48"
dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
from {username='slave'} at 127.0.0.1:39626
I20260808 20:12:54.696879 11311 raft_consensus.cc:482] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:54.697178 11311 raft_consensus.cc:3037] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:54.701668 11311 raft_consensus.cc:504] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } } peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } }
I20260808 20:12:54.703331 11311 leader_election.cc:288] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 2 election: Requested vote from peers 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783), 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075)
I20260808 20:12:54.703963 11532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4e13f27b9d4abbbda7599371e4ad48" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "124d7af2a136423c9165edf5e240ec12"
I20260808 20:12:54.704252 11417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4e13f27b9d4abbbda7599371e4ad48" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2fa933dd5405492aa709133798d4ba98"
I20260808 20:12:54.704515 11532 raft_consensus.cc:3037] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:54.704769 11417 raft_consensus.cc:3032] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:12:54.705109 11417 raft_consensus.cc:729] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2fa933dd5405492aa709133798d4ba98, State: Running, Role: LEADER
I20260808 20:12:54.705818 11417 consensus_queue.cc:260] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } } peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } }
I20260808 20:12:54.707048 11417 raft_consensus.cc:3037] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:54.710505 11532 raft_consensus.cc:2454] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb8edc2375f449ac806aebab2d5fd6a8 in term 2.
I20260808 20:12:54.711436 11257 leader_election.cc:302] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, bb8edc2375f449ac806aebab2d5fd6a8; no voters:
I20260808 20:12:54.712035 11664 raft_consensus.cc:2781] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:12:54.712374 11664 raft_consensus.cc:686] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Becoming Leader. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Running, Role: LEADER
I20260808 20:12:54.712685 11417 raft_consensus.cc:2454] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb8edc2375f449ac806aebab2d5fd6a8 in term 2.
I20260808 20:12:54.713109 11664 consensus_queue.cc:237] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } } peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } }
I20260808 20:12:54.720618 11163 catalog_manager.cc:5495] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 reported cstate change: term changed from 1 to 2, leader changed from 2fa933dd5405492aa709133798d4ba98 (127.10.24.2) to bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:54.777709 11674 consensus_queue.cc:1035] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:54.783809 11683 raft_consensus.cc:979] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12: : Instructing follower bb8edc2375f449ac806aebab2d5fd6a8 to start an election
I20260808 20:12:54.784264 11677 raft_consensus.cc:1067] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Signalling peer bb8edc2375f449ac806aebab2d5fd6a8 to start an election
I20260808 20:12:54.786047 11310 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d6a5465239d74c75bfae9807d91bccba"
dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
from {username='slave'} at 127.0.0.1:39614
I20260808 20:12:54.786587 11310 raft_consensus.cc:482] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:54.786955 11310 raft_consensus.cc:3037] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:54.790500 11683 consensus_queue.cc:1035] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:54.792372 11683 consensus_queue.cc:1035] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:54.792981 11310 raft_consensus.cc:504] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.795424 11418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a5465239d74c75bfae9807d91bccba" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2fa933dd5405492aa709133798d4ba98"
I20260808 20:12:54.796450 11532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a5465239d74c75bfae9807d91bccba" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "124d7af2a136423c9165edf5e240ec12"
I20260808 20:12:54.797060 11532 raft_consensus.cc:3032] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:12:54.797351 11532 raft_consensus.cc:729] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Running, Role: LEADER
I20260808 20:12:54.798049 11532 consensus_queue.cc:260] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.799837 11532 raft_consensus.cc:3037] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:54.800688 11310 leader_election.cc:288] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 2 election: Requested vote from peers 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075), 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783)
I20260808 20:12:54.806322 11683 consensus_queue.cc:1035] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:54.809952 11532 raft_consensus.cc:2454] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb8edc2375f449ac806aebab2d5fd6a8 in term 2.
I20260808 20:12:54.810779 11257 leader_election.cc:302] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, bb8edc2375f449ac806aebab2d5fd6a8; no voters: 2fa933dd5405492aa709133798d4ba98
I20260808 20:12:54.811432 11664 raft_consensus.cc:2781] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:12:54.811896 11664 raft_consensus.cc:686] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Becoming Leader. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Running, Role: LEADER
I20260808 20:12:54.812788 11664 consensus_queue.cc:237] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.817482 11605 consensus_queue.cc:1035] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:54.818055 11674 raft_consensus.cc:979] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12: : Instructing follower 2fa933dd5405492aa709133798d4ba98 to start an election
I20260808 20:12:54.818459 11674 raft_consensus.cc:1067] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Signalling peer 2fa933dd5405492aa709133798d4ba98 to start an election
I20260808 20:12:54.820706 11417 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "304a2461fac34933aa3a07f54b875889"
dest_uuid: "2fa933dd5405492aa709133798d4ba98"
from {username='slave'} at 127.0.0.1:47798
I20260808 20:12:54.821259 11417 raft_consensus.cc:482] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:54.821641 11417 raft_consensus.cc:3037] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:54.824491 11160 catalog_manager.cc:5495] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 reported cstate change: term changed from 1 to 2, leader changed from 124d7af2a136423c9165edf5e240ec12 (127.10.24.3) to bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:54.830052 11656 consensus_queue.cc:1035] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:54.831389 11656 consensus_queue.cc:1035] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:54.832911 11417 raft_consensus.cc:504] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.840353 11417 leader_election.cc:288] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 2 election: Requested vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783)
I20260808 20:12:54.840344 11311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304a2461fac34933aa3a07f54b875889" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
I20260808 20:12:54.840955 11532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304a2461fac34933aa3a07f54b875889" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "124d7af2a136423c9165edf5e240ec12"
I20260808 20:12:54.841137 11311 raft_consensus.cc:3037] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:54.841544 11532 raft_consensus.cc:3032] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:12:54.841856 11532 raft_consensus.cc:729] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Running, Role: LEADER
I20260808 20:12:54.842592 11532 consensus_queue.cc:260] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.843829 11532 raft_consensus.cc:3037] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:54.847599 11311 raft_consensus.cc:2454] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fa933dd5405492aa709133798d4ba98 in term 2.
I20260808 20:12:54.848623 11365 leader_election.cc:302] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fa933dd5405492aa709133798d4ba98, bb8edc2375f449ac806aebab2d5fd6a8; no voters:
I20260808 20:12:54.849432 11656 raft_consensus.cc:2781] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:12:54.849908 11656 raft_consensus.cc:686] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 2 LEADER]: Becoming Leader. State: Replica: 2fa933dd5405492aa709133798d4ba98, State: Running, Role: LEADER
I20260808 20:12:54.850384 11532 raft_consensus.cc:2454] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fa933dd5405492aa709133798d4ba98 in term 2.
I20260808 20:12:54.850680 11656 consensus_queue.cc:237] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.860000 11656 consensus_queue.cc:1035] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:54.859735 11158 catalog_manager.cc:5495] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 reported cstate change: term changed from 1 to 2, leader changed from 124d7af2a136423c9165edf5e240ec12 (127.10.24.3) to 2fa933dd5405492aa709133798d4ba98 (127.10.24.2). New cstate: current_term: 2 leader_uuid: "2fa933dd5405492aa709133798d4ba98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:54.921108 11683 consensus_queue.cc:1035] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:54.932235 11677 consensus_queue.cc:1035] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:54.938638 11683 raft_consensus.cc:979] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12: : Instructing follower 2fa933dd5405492aa709133798d4ba98 to start an election
I20260808 20:12:54.939015 11677 raft_consensus.cc:1067] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Signalling peer 2fa933dd5405492aa709133798d4ba98 to start an election
I20260808 20:12:54.940338 11417 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "089736c28b834bd5a121f818bcb364d7"
dest_uuid: "2fa933dd5405492aa709133798d4ba98"
from {username='slave'} at 127.0.0.1:47798
I20260808 20:12:54.940745 11417 raft_consensus.cc:482] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:54.941035 11417 raft_consensus.cc:3037] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:54.945293 11417 raft_consensus.cc:504] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.946702 11417 leader_election.cc:288] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 2 election: Requested vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783)
I20260808 20:12:54.947476 11310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089736c28b834bd5a121f818bcb364d7" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
I20260808 20:12:54.947762 11530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089736c28b834bd5a121f818bcb364d7" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "124d7af2a136423c9165edf5e240ec12"
I20260808 20:12:54.948031 11310 raft_consensus.cc:3037] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:54.948280 11530 raft_consensus.cc:3032] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:12:54.948563 11530 raft_consensus.cc:729] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Running, Role: LEADER
I20260808 20:12:54.949317 11530 consensus_queue.cc:260] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.950464 11530 raft_consensus.cc:3037] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:54.953927 11310 raft_consensus.cc:2454] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fa933dd5405492aa709133798d4ba98 in term 2.
I20260808 20:12:54.954903 11365 leader_election.cc:302] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fa933dd5405492aa709133798d4ba98, bb8edc2375f449ac806aebab2d5fd6a8; no voters:
I20260808 20:12:54.955478 11656 raft_consensus.cc:2781] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:12:54.955829 11656 raft_consensus.cc:686] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98 [term 2 LEADER]: Becoming Leader. State: Replica: 2fa933dd5405492aa709133798d4ba98, State: Running, Role: LEADER
I20260808 20:12:54.955895 11530 raft_consensus.cc:2454] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fa933dd5405492aa709133798d4ba98 in term 2.
I20260808 20:12:54.956563 11656 consensus_queue.cc:237] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:54.961895 11160 catalog_manager.cc:5495] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98 reported cstate change: term changed from 1 to 2, leader changed from 124d7af2a136423c9165edf5e240ec12 (127.10.24.3) to 2fa933dd5405492aa709133798d4ba98 (127.10.24.2). New cstate: current_term: 2 leader_uuid: "2fa933dd5405492aa709133798d4ba98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:55.020193 11683 raft_consensus.cc:979] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12: : Instructing follower bb8edc2375f449ac806aebab2d5fd6a8 to start an election
I20260808 20:12:55.020567 11677 raft_consensus.cc:1067] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Signalling peer bb8edc2375f449ac806aebab2d5fd6a8 to start an election
I20260808 20:12:55.021729 11310 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "261b728800574016968c1083c414e27f"
dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
from {username='slave'} at 127.0.0.1:39614
I20260808 20:12:55.022176 11310 raft_consensus.cc:482] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:55.022439 11310 raft_consensus.cc:3037] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:55.026899 11310 raft_consensus.cc:504] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:55.028242 11310 leader_election.cc:288] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 2 election: Requested vote from peers 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075), 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783)
I20260808 20:12:55.029012 11417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "261b728800574016968c1083c414e27f" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2fa933dd5405492aa709133798d4ba98"
I20260808 20:12:55.029541 11417 raft_consensus.cc:3037] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:55.029747 11530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "261b728800574016968c1083c414e27f" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "124d7af2a136423c9165edf5e240ec12"
I20260808 20:12:55.030319 11530 raft_consensus.cc:3032] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:12:55.030617 11530 raft_consensus.cc:729] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Running, Role: LEADER
I20260808 20:12:55.031395 11530 consensus_queue.cc:260] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:55.032547 11530 raft_consensus.cc:3037] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:55.035498 11417 raft_consensus.cc:2454] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb8edc2375f449ac806aebab2d5fd6a8 in term 2.
I20260808 20:12:55.036507 11257 leader_election.cc:302] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fa933dd5405492aa709133798d4ba98, bb8edc2375f449ac806aebab2d5fd6a8; no voters:
I20260808 20:12:55.037271 11664 raft_consensus.cc:2781] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:12:55.037624 11664 raft_consensus.cc:686] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Becoming Leader. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Running, Role: LEADER
I20260808 20:12:55.038089 11530 raft_consensus.cc:2454] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb8edc2375f449ac806aebab2d5fd6a8 in term 2.
I20260808 20:12:55.038265 11664 consensus_queue.cc:237] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:55.043646 11160 catalog_manager.cc:5495] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 reported cstate change: term changed from 1 to 2, leader changed from 124d7af2a136423c9165edf5e240ec12 (127.10.24.3) to bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:55.071198 11683 raft_consensus.cc:979] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12: : Instructing follower bb8edc2375f449ac806aebab2d5fd6a8 to start an election
I20260808 20:12:55.071558 11677 raft_consensus.cc:1067] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Signalling peer bb8edc2375f449ac806aebab2d5fd6a8 to start an election
I20260808 20:12:55.072777 11310 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "44506b0c529c45efbee5ac3003ef6f42"
dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
from {username='slave'} at 127.0.0.1:39614
I20260808 20:12:55.073335 11310 raft_consensus.cc:482] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:55.073683 11310 raft_consensus.cc:3037] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:55.078032 11310 raft_consensus.cc:504] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:55.079335 11310 leader_election.cc:288] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 2 election: Requested vote from peers 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075), 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783)
I20260808 20:12:55.080118 11417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44506b0c529c45efbee5ac3003ef6f42" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2fa933dd5405492aa709133798d4ba98"
I20260808 20:12:55.080256 11530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44506b0c529c45efbee5ac3003ef6f42" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "124d7af2a136423c9165edf5e240ec12"
I20260808 20:12:55.080683 11417 raft_consensus.cc:3037] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:55.080763 11530 raft_consensus.cc:3032] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:12:55.081121 11530 raft_consensus.cc:729] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Running, Role: LEADER
I20260808 20:12:55.081841 11530 consensus_queue.cc:260] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:55.083143 11530 raft_consensus.cc:3037] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:55.087019 11677 consensus_queue.cc:1035] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:55.087121 11417 raft_consensus.cc:2454] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb8edc2375f449ac806aebab2d5fd6a8 in term 2.
I20260808 20:12:55.088176 11257 leader_election.cc:302] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fa933dd5405492aa709133798d4ba98, bb8edc2375f449ac806aebab2d5fd6a8; no voters:
I20260808 20:12:55.088881 11664 raft_consensus.cc:2781] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:12:55.089344 11664 raft_consensus.cc:686] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Becoming Leader. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Running, Role: LEADER
I20260808 20:12:55.090245 11677 consensus_queue.cc:1035] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:12:55.090097 11664 consensus_queue.cc:237] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:55.094679 11530 raft_consensus.cc:2454] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb8edc2375f449ac806aebab2d5fd6a8 in term 2.
I20260808 20:12:55.098670 11163 catalog_manager.cc:5495] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 reported cstate change: term changed from 1 to 2, leader changed from 124d7af2a136423c9165edf5e240ec12 (127.10.24.3) to bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:55.102803 11683 raft_consensus.cc:979] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12: : Instructing follower bb8edc2375f449ac806aebab2d5fd6a8 to start an election
I20260808 20:12:55.103248 11674 raft_consensus.cc:1067] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Signalling peer bb8edc2375f449ac806aebab2d5fd6a8 to start an election
I20260808 20:12:55.104941 11311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "941577432a4440b981675cc78e482325"
dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
from {username='slave'} at 127.0.0.1:39614
I20260808 20:12:55.105638 11311 raft_consensus.cc:482] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:55.105984 11311 raft_consensus.cc:3037] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:55.111862 11311 raft_consensus.cc:504] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:55.113292 11311 leader_election.cc:288] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 2 election: Requested vote from peers 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075), 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783)
I20260808 20:12:55.114118 11417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941577432a4440b981675cc78e482325" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2fa933dd5405492aa709133798d4ba98"
I20260808 20:12:55.114436 11530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941577432a4440b981675cc78e482325" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "124d7af2a136423c9165edf5e240ec12"
I20260808 20:12:55.114672 11417 raft_consensus.cc:3037] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:55.114965 11530 raft_consensus.cc:3032] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:12:55.115283 11530 raft_consensus.cc:729] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Running, Role: LEADER
I20260808 20:12:55.115972 11530 consensus_queue.cc:260] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:55.117165 11530 raft_consensus.cc:3037] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:55.120527 11417 raft_consensus.cc:2454] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb8edc2375f449ac806aebab2d5fd6a8 in term 2.
I20260808 20:12:55.121461 11257 leader_election.cc:302] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fa933dd5405492aa709133798d4ba98, bb8edc2375f449ac806aebab2d5fd6a8; no voters:
I20260808 20:12:55.122071 11664 raft_consensus.cc:2781] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:12:55.122524 11664 raft_consensus.cc:686] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Becoming Leader. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Running, Role: LEADER
I20260808 20:12:55.122725 11530 raft_consensus.cc:2454] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb8edc2375f449ac806aebab2d5fd6a8 in term 2.
I20260808 20:12:55.123296 11664 consensus_queue.cc:237] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:55.127707 11417 raft_consensus.cc:1261] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Refusing update from remote peer bb8edc2375f449ac806aebab2d5fd6a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:55.128772 11664 consensus_queue.cc:1035] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:55.129282 11163 catalog_manager.cc:5495] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 reported cstate change: term changed from 1 to 2, leader changed from 124d7af2a136423c9165edf5e240ec12 (127.10.24.3) to bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:55.137583 11530 raft_consensus.cc:1261] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Refusing update from remote peer bb8edc2375f449ac806aebab2d5fd6a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:55.138985 11710 consensus_queue.cc:1035] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:55.218353 11417 raft_consensus.cc:3037] T d6a5465239d74c75bfae9807d91bccba P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:12:55.224159 11417 raft_consensus.cc:1261] T d6a5465239d74c75bfae9807d91bccba P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Refusing update from remote peer bb8edc2375f449ac806aebab2d5fd6a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:55.225296 11710 consensus_queue.cc:1035] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:55.232293 11530 raft_consensus.cc:1261] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Refusing update from remote peer bb8edc2375f449ac806aebab2d5fd6a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:55.233594 11664 consensus_queue.cc:1035] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:55.338680 11311 raft_consensus.cc:1261] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Refusing update from remote peer 2fa933dd5405492aa709133798d4ba98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:55.339764 11656 consensus_queue.cc:1035] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:55.346388 11530 raft_consensus.cc:1261] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Refusing update from remote peer 2fa933dd5405492aa709133798d4ba98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:55.347656 11605 consensus_queue.cc:1035] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:55.350621 11532 raft_consensus.cc:1261] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Refusing update from remote peer 2fa933dd5405492aa709133798d4ba98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:55.351954 11659 consensus_queue.cc:1035] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:55.359014 11311 raft_consensus.cc:1261] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Refusing update from remote peer 2fa933dd5405492aa709133798d4ba98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:55.360785 11712 consensus_queue.cc:1035] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:55.433174 11417 raft_consensus.cc:1261] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Refusing update from remote peer bb8edc2375f449ac806aebab2d5fd6a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:55.434163 11664 consensus_queue.cc:1035] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:55.440819 11532 raft_consensus.cc:1261] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Refusing update from remote peer bb8edc2375f449ac806aebab2d5fd6a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:55.442175 11664 consensus_queue.cc:1035] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:55.532285 11532 raft_consensus.cc:1261] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Refusing update from remote peer bb8edc2375f449ac806aebab2d5fd6a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:55.533377 11664 consensus_queue.cc:1035] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:55.540583 11417 raft_consensus.cc:1261] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Refusing update from remote peer bb8edc2375f449ac806aebab2d5fd6a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:55.542100 11711 consensus_queue.cc:1035] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:55.705740 11532 raft_consensus.cc:1261] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Refusing update from remote peer bb8edc2375f449ac806aebab2d5fd6a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:55.706938 11711 consensus_queue.cc:1035] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:55.715221 11417 raft_consensus.cc:1261] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Refusing update from remote peer bb8edc2375f449ac806aebab2d5fd6a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:12:55.716547 11711 consensus_queue.cc:1035] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:12:57.649071 10336 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20260808 20:12:57.649668 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:12:57.668735 11311 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44506b0c529c45efbee5ac3003ef6f42"
dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
mode: GRACEFUL
new_leader_uuid: "124d7af2a136423c9165edf5e240ec12"
from {username='slave'} at 127.0.0.1:39632
I20260808 20:12:57.669384 11311 raft_consensus.cc:595] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Received request to transfer leadership to TS 124d7af2a136423c9165edf5e240ec12
I20260808 20:12:57.671435 11311 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e4e13f27b9d4abbbda7599371e4ad48"
dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
mode: GRACEFUL
new_leader_uuid: "124d7af2a136423c9165edf5e240ec12"
from {username='slave'} at 127.0.0.1:39632
I20260808 20:12:57.671826 11311 raft_consensus.cc:595] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Received request to transfer leadership to TS 124d7af2a136423c9165edf5e240ec12
I20260808 20:12:57.672578 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260808 20:12:57.672942 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:12:57.739737 11664 raft_consensus.cc:979] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8: : Instructing follower 124d7af2a136423c9165edf5e240ec12 to start an election
I20260808 20:12:57.740170 11715 raft_consensus.cc:1067] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Signalling peer 124d7af2a136423c9165edf5e240ec12 to start an election
I20260808 20:12:57.741460 11532 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e4e13f27b9d4abbbda7599371e4ad48"
dest_uuid: "124d7af2a136423c9165edf5e240ec12"
from {username='slave'} at 127.0.0.1:52856
I20260808 20:12:57.741890 11532 raft_consensus.cc:482] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:57.742159 11532 raft_consensus.cc:3037] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:12:57.746399 11532 raft_consensus.cc:504] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } } peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } }
I20260808 20:12:57.747706 11532 leader_election.cc:288] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 3 election: Requested vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075)
I20260808 20:12:57.748520 11311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4e13f27b9d4abbbda7599371e4ad48" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
I20260808 20:12:57.748672 11418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4e13f27b9d4abbbda7599371e4ad48" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2fa933dd5405492aa709133798d4ba98"
I20260808 20:12:57.749099 11311 raft_consensus.cc:3032] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:12:57.749231 11418 raft_consensus.cc:3037] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:12:57.749408 11311 raft_consensus.cc:729] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Running, Role: LEADER
I20260808 20:12:57.750106 11311 consensus_queue.cc:260] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } } peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } }
I20260808 20:12:57.751026 11311 raft_consensus.cc:3037] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:12:57.753293 11418 raft_consensus.cc:2454] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 3.
I20260808 20:12:57.754139 11475 leader_election.cc:302] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, 2fa933dd5405492aa709133798d4ba98; no voters:
I20260808 20:12:57.754712 11612 raft_consensus.cc:2781] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:12:57.755481 11311 raft_consensus.cc:2454] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 3.
I20260808 20:12:57.755606 11612 raft_consensus.cc:686] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [term 3 LEADER]: Becoming Leader. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Running, Role: LEADER
I20260808 20:12:57.756408 11612 consensus_queue.cc:237] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } } peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } }
I20260808 20:12:57.762552 11160 catalog_manager.cc:5495] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 reported cstate change: term changed from 2 to 3, leader changed from bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1) to 124d7af2a136423c9165edf5e240ec12 (127.10.24.3). New cstate: current_term: 3 leader_uuid: "124d7af2a136423c9165edf5e240ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: UNKNOWN } } }
I20260808 20:12:57.770466 11664 raft_consensus.cc:979] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8: : Instructing follower 124d7af2a136423c9165edf5e240ec12 to start an election
I20260808 20:12:57.770869 11664 raft_consensus.cc:1067] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Signalling peer 124d7af2a136423c9165edf5e240ec12 to start an election
I20260808 20:12:57.772305 11532 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "44506b0c529c45efbee5ac3003ef6f42"
dest_uuid: "124d7af2a136423c9165edf5e240ec12"
from {username='slave'} at 127.0.0.1:52856
I20260808 20:12:57.772749 11532 raft_consensus.cc:482] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:12:57.773108 11532 raft_consensus.cc:3037] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:12:57.777195 11532 raft_consensus.cc:504] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:57.778609 11532 leader_election.cc:288] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 3 election: Requested vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075)
I20260808 20:12:57.779243 11311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44506b0c529c45efbee5ac3003ef6f42" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
I20260808 20:12:57.779561 11418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44506b0c529c45efbee5ac3003ef6f42" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2fa933dd5405492aa709133798d4ba98"
I20260808 20:12:57.779804 11311 raft_consensus.cc:3032] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:12:57.780084 11418 raft_consensus.cc:3037] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:12:57.780100 11311 raft_consensus.cc:729] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Running, Role: LEADER
I20260808 20:12:57.780961 11311 consensus_queue.cc:260] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:57.782110 11311 raft_consensus.cc:3037] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:12:57.786175 11418 raft_consensus.cc:2454] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 3.
I20260808 20:12:57.787102 11475 leader_election.cc:302] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, 2fa933dd5405492aa709133798d4ba98; no voters:
I20260808 20:12:57.787796 11612 raft_consensus.cc:2781] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:12:57.787905 11311 raft_consensus.cc:2454] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 3.
I20260808 20:12:57.788275 11612 raft_consensus.cc:686] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 3 LEADER]: Becoming Leader. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Running, Role: LEADER
I20260808 20:12:57.789069 11612 consensus_queue.cc:237] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:12:57.795118 11160 catalog_manager.cc:5495] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 reported cstate change: term changed from 2 to 3, leader changed from bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1) to 124d7af2a136423c9165edf5e240ec12 (127.10.24.3). New cstate: current_term: 3 leader_uuid: "124d7af2a136423c9165edf5e240ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: HEALTHY } } }
I20260808 20:12:58.194284 11311 raft_consensus.cc:1261] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 FOLLOWER]: Refusing update from remote peer 124d7af2a136423c9165edf5e240ec12: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:12:58.195369 11719 consensus_queue.cc:1035] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:12:58.208319 11418 raft_consensus.cc:1261] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98 [term 3 FOLLOWER]: Refusing update from remote peer 124d7af2a136423c9165edf5e240ec12: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:12:58.211429 11719 consensus_queue.cc:1035] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:12:58.213217 11311 raft_consensus.cc:1261] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 FOLLOWER]: Refusing update from remote peer 124d7af2a136423c9165edf5e240ec12: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:12:58.214591 11719 consensus_queue.cc:1035] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:12:58.226882 11418 raft_consensus.cc:1261] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 3 FOLLOWER]: Refusing update from remote peer 124d7af2a136423c9165edf5e240ec12: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:12:58.228119 11737 consensus_queue.cc:1035] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:12:58.673705 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:12:58.678767 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:12:58.679139 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:12:59.679884 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:12:59.685067 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:12:59.685452 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:00.686229 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:00.692145 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:13:00.692638 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:01.693440 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:01.699271 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:13:01.699656 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:02.700739 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:02.706266 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:13:02.706636 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:03.707335 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:03.713433 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:13:03.713909 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:04.714741 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:04.719703 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:13:04.720085 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:05.720836 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:05.725822 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:13:05.726213 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:06.726977 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:06.733167 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:13:06.733597 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:07.760342 10336 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20260808 20:13:07.765532 11530 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e4e13f27b9d4abbbda7599371e4ad48"
dest_uuid: "124d7af2a136423c9165edf5e240ec12"
mode: GRACEFUL
new_leader_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
from {username='slave'} at 127.0.0.1:52868
I20260808 20:13:07.766182 11530 raft_consensus.cc:595] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [term 3 LEADER]: Received request to transfer leadership to TS bb8edc2375f449ac806aebab2d5fd6a8
I20260808 20:13:07.776598 11530 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44506b0c529c45efbee5ac3003ef6f42"
dest_uuid: "124d7af2a136423c9165edf5e240ec12"
mode: GRACEFUL
new_leader_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
from {username='slave'} at 127.0.0.1:52868
I20260808 20:13:07.777058 11530 raft_consensus.cc:595] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 3 LEADER]: Received request to transfer leadership to TS bb8edc2375f449ac806aebab2d5fd6a8
I20260808 20:13:07.781116 11418 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2719138be134d1382e8c595a0275d7f"
dest_uuid: "2fa933dd5405492aa709133798d4ba98"
mode: GRACEFUL
new_leader_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
from {username='slave'} at 127.0.0.1:47822
I20260808 20:13:07.781487 11418 raft_consensus.cc:595] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [term 1 LEADER]: Received request to transfer leadership to TS bb8edc2375f449ac806aebab2d5fd6a8
I20260808 20:13:07.785382 11418 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f616636ffca943a7a1731ee069940307"
dest_uuid: "2fa933dd5405492aa709133798d4ba98"
mode: GRACEFUL
new_leader_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
from {username='slave'} at 127.0.0.1:47822
I20260808 20:13:07.785766 11418 raft_consensus.cc:595] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [term 1 LEADER]: Received request to transfer leadership to TS bb8edc2375f449ac806aebab2d5fd6a8
I20260808 20:13:07.789536 11418 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "304a2461fac34933aa3a07f54b875889"
dest_uuid: "2fa933dd5405492aa709133798d4ba98"
mode: GRACEFUL
new_leader_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
from {username='slave'} at 127.0.0.1:47822
I20260808 20:13:07.789907 11418 raft_consensus.cc:595] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 2 LEADER]: Received request to transfer leadership to TS bb8edc2375f449ac806aebab2d5fd6a8
I20260808 20:13:07.799914 11712 raft_consensus.cc:979] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98: : Instructing follower bb8edc2375f449ac806aebab2d5fd6a8 to start an election
I20260808 20:13:07.800344 11757 raft_consensus.cc:1067] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [term 1 LEADER]: Signalling peer bb8edc2375f449ac806aebab2d5fd6a8 to start an election
I20260808 20:13:07.801702 11311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f2719138be134d1382e8c595a0275d7f"
dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
from {username='slave'} at 127.0.0.1:39626
I20260808 20:13:07.802090 11311 raft_consensus.cc:482] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:07.802402 11311 raft_consensus.cc:3037] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:07.807516 11311 raft_consensus.cc:504] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:07.808939 11311 leader_election.cc:288] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 2 election: Requested vote from peers 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075), 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783)
I20260808 20:13:07.809674 11418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2719138be134d1382e8c595a0275d7f" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2fa933dd5405492aa709133798d4ba98"
I20260808 20:13:07.810400 11418 raft_consensus.cc:3032] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:07.810372 11530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2719138be134d1382e8c595a0275d7f" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "124d7af2a136423c9165edf5e240ec12"
I20260808 20:13:07.810807 11418 raft_consensus.cc:729] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2fa933dd5405492aa709133798d4ba98, State: Running, Role: LEADER
I20260808 20:13:07.811036 11530 raft_consensus.cc:3037] T f2719138be134d1382e8c595a0275d7f P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:07.811561 11418 consensus_queue.cc:260] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:07.813001 11418 raft_consensus.cc:3037] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:07.818728 11530 raft_consensus.cc:2454] T f2719138be134d1382e8c595a0275d7f P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb8edc2375f449ac806aebab2d5fd6a8 in term 2.
I20260808 20:13:07.819665 11418 raft_consensus.cc:2454] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb8edc2375f449ac806aebab2d5fd6a8 in term 2.
I20260808 20:13:07.819883 11257 leader_election.cc:302] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, bb8edc2375f449ac806aebab2d5fd6a8; no voters:
I20260808 20:13:07.820675 11754 raft_consensus.cc:2781] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:07.821548 11754 raft_consensus.cc:686] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Becoming Leader. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Running, Role: LEADER
I20260808 20:13:07.822227 11754 consensus_queue.cc:237] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:07.829137 11160 catalog_manager.cc:5495] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8 reported cstate change: term changed from 1 to 2, leader changed from 2fa933dd5405492aa709133798d4ba98 (127.10.24.2) to bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:07.883682 11712 raft_consensus.cc:979] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98: : Instructing follower bb8edc2375f449ac806aebab2d5fd6a8 to start an election
I20260808 20:13:07.884058 11712 raft_consensus.cc:1067] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [term 1 LEADER]: Signalling peer bb8edc2375f449ac806aebab2d5fd6a8 to start an election
I20260808 20:13:07.885456 11310 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f616636ffca943a7a1731ee069940307"
dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
from {username='slave'} at 127.0.0.1:39626
I20260808 20:13:07.885986 11310 raft_consensus.cc:482] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:07.886304 11310 raft_consensus.cc:3037] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:07.887588 11755 raft_consensus.cc:979] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12: : Instructing follower bb8edc2375f449ac806aebab2d5fd6a8 to start an election
I20260808 20:13:07.888103 11768 raft_consensus.cc:1067] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 3 LEADER]: Signalling peer bb8edc2375f449ac806aebab2d5fd6a8 to start an election
I20260808 20:13:07.889547 11311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "44506b0c529c45efbee5ac3003ef6f42"
dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
from {username='slave'} at 127.0.0.1:39614
I20260808 20:13:07.890014 11311 raft_consensus.cc:482] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:07.890345 11311 raft_consensus.cc:3037] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:13:07.893221 11310 raft_consensus.cc:504] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:07.895103 11310 leader_election.cc:288] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 2 election: Requested vote from peers 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075), 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783)
I20260808 20:13:07.895669 11418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f616636ffca943a7a1731ee069940307" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2fa933dd5405492aa709133798d4ba98"
I20260808 20:13:07.896013 11530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f616636ffca943a7a1731ee069940307" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "124d7af2a136423c9165edf5e240ec12"
I20260808 20:13:07.896231 11418 raft_consensus.cc:3032] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:07.896590 11530 raft_consensus.cc:3037] T f616636ffca943a7a1731ee069940307 P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:07.896234 11311 raft_consensus.cc:504] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:07.896710 11418 raft_consensus.cc:729] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2fa933dd5405492aa709133798d4ba98, State: Running, Role: LEADER
I20260808 20:13:07.897476 11418 consensus_queue.cc:260] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:07.898080 11311 leader_election.cc:288] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 4 election: Requested vote from peers 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075), 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783)
I20260808 20:13:07.898638 11418 raft_consensus.cc:3037] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:07.899196 11417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44506b0c529c45efbee5ac3003ef6f42" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2fa933dd5405492aa709133798d4ba98"
I20260808 20:13:07.899467 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44506b0c529c45efbee5ac3003ef6f42" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "124d7af2a136423c9165edf5e240ec12"
I20260808 20:13:07.899714 11417 raft_consensus.cc:3037] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:13:07.902119 11531 raft_consensus.cc:3032] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:13:07.902369 11531 raft_consensus.cc:729] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Running, Role: LEADER
I20260808 20:13:07.903052 11531 consensus_queue.cc:260] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:07.903967 11531 raft_consensus.cc:3037] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:13:07.905113 11418 raft_consensus.cc:2454] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb8edc2375f449ac806aebab2d5fd6a8 in term 2.
I20260808 20:13:07.905988 11257 leader_election.cc:302] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fa933dd5405492aa709133798d4ba98, bb8edc2375f449ac806aebab2d5fd6a8; no voters:
I20260808 20:13:07.906474 11417 raft_consensus.cc:2454] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb8edc2375f449ac806aebab2d5fd6a8 in term 4.
I20260808 20:13:07.906697 11754 raft_consensus.cc:2781] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:07.907202 11754 raft_consensus.cc:686] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Becoming Leader. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Running, Role: LEADER
I20260808 20:13:07.907477 11257 leader_election.cc:302] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fa933dd5405492aa709133798d4ba98, bb8edc2375f449ac806aebab2d5fd6a8; no voters:
I20260808 20:13:07.908064 11754 consensus_queue.cc:237] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:07.909480 11758 raft_consensus.cc:2781] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:13:07.909958 11758 raft_consensus.cc:686] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [term 4 LEADER]: Becoming Leader. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Running, Role: LEADER
I20260808 20:13:07.910040 11531 raft_consensus.cc:2454] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb8edc2375f449ac806aebab2d5fd6a8 in term 4.
I20260808 20:13:07.910874 11758 consensus_queue.cc:237] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:07.912065 11530 raft_consensus.cc:2454] T f616636ffca943a7a1731ee069940307 P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb8edc2375f449ac806aebab2d5fd6a8 in term 2.
I20260808 20:13:07.920673 11160 catalog_manager.cc:5495] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8 reported cstate change: term changed from 1 to 2, leader changed from 2fa933dd5405492aa709133798d4ba98 (127.10.24.2) to bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:07.930361 11160 catalog_manager.cc:5495] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 reported cstate change: term changed from 3 to 4, leader changed from 124d7af2a136423c9165edf5e240ec12 (127.10.24.3) to bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1). New cstate: current_term: 4 leader_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:08.149432 11757 raft_consensus.cc:979] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98: : Instructing follower bb8edc2375f449ac806aebab2d5fd6a8 to start an election
I20260808 20:13:08.149837 11712 raft_consensus.cc:1067] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 2 LEADER]: Signalling peer bb8edc2375f449ac806aebab2d5fd6a8 to start an election
I20260808 20:13:08.150906 11311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "304a2461fac34933aa3a07f54b875889"
dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
from {username='slave'} at 127.0.0.1:39626
I20260808 20:13:08.151317 11311 raft_consensus.cc:482] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:08.151563 11311 raft_consensus.cc:3037] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:08.155516 11311 raft_consensus.cc:504] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:08.156711 11311 leader_election.cc:288] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 3 election: Requested vote from peers 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075), 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783)
I20260808 20:13:08.157495 11417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304a2461fac34933aa3a07f54b875889" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2fa933dd5405492aa709133798d4ba98"
I20260808 20:13:08.157641 11530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304a2461fac34933aa3a07f54b875889" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "124d7af2a136423c9165edf5e240ec12"
I20260808 20:13:08.158073 11417 raft_consensus.cc:3032] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:13:08.158133 11530 raft_consensus.cc:3037] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:08.158447 11417 raft_consensus.cc:729] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2fa933dd5405492aa709133798d4ba98, State: Running, Role: LEADER
I20260808 20:13:08.159250 11417 consensus_queue.cc:260] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:08.160521 11417 raft_consensus.cc:3037] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:08.164422 11530 raft_consensus.cc:2454] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb8edc2375f449ac806aebab2d5fd6a8 in term 3.
I20260808 20:13:08.165419 11257 leader_election.cc:302] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, bb8edc2375f449ac806aebab2d5fd6a8; no voters:
I20260808 20:13:08.166141 11754 raft_consensus.cc:2781] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:13:08.166384 11417 raft_consensus.cc:2454] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb8edc2375f449ac806aebab2d5fd6a8 in term 3.
I20260808 20:13:08.166538 11754 raft_consensus.cc:686] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 LEADER]: Becoming Leader. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Running, Role: LEADER
I20260808 20:13:08.167578 11754 consensus_queue.cc:237] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:08.174333 11160 catalog_manager.cc:5495] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 reported cstate change: term changed from 2 to 3, leader changed from 2fa933dd5405492aa709133798d4ba98 (127.10.24.2) to bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1). New cstate: current_term: 3 leader_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:08.245383 11755 raft_consensus.cc:979] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12: : Instructing follower bb8edc2375f449ac806aebab2d5fd6a8 to start an election
I20260808 20:13:08.245805 11768 raft_consensus.cc:1067] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [term 3 LEADER]: Signalling peer bb8edc2375f449ac806aebab2d5fd6a8 to start an election
I20260808 20:13:08.247066 11311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e4e13f27b9d4abbbda7599371e4ad48"
dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
from {username='slave'} at 127.0.0.1:39614
I20260808 20:13:08.247490 11311 raft_consensus.cc:482] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:08.247741 11311 raft_consensus.cc:3037] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:13:08.251749 11311 raft_consensus.cc:504] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } } peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } }
I20260808 20:13:08.253046 11311 leader_election.cc:288] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 4 election: Requested vote from peers 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783), 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075)
I20260808 20:13:08.253738 11530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4e13f27b9d4abbbda7599371e4ad48" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "124d7af2a136423c9165edf5e240ec12"
I20260808 20:13:08.253968 11417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4e13f27b9d4abbbda7599371e4ad48" candidate_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2fa933dd5405492aa709133798d4ba98"
I20260808 20:13:08.254354 11530 raft_consensus.cc:3032] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:13:08.254427 11417 raft_consensus.cc:3037] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:13:08.254698 11530 raft_consensus.cc:729] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Running, Role: LEADER
I20260808 20:13:08.255484 11530 consensus_queue.cc:260] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } } peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } }
I20260808 20:13:08.256716 11530 raft_consensus.cc:3037] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:13:08.260257 11417 raft_consensus.cc:2454] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb8edc2375f449ac806aebab2d5fd6a8 in term 4.
I20260808 20:13:08.261142 11257 leader_election.cc:302] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fa933dd5405492aa709133798d4ba98, bb8edc2375f449ac806aebab2d5fd6a8; no voters:
I20260808 20:13:08.261832 11754 raft_consensus.cc:2781] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:13:08.262230 11754 raft_consensus.cc:686] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 4 LEADER]: Becoming Leader. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Running, Role: LEADER
I20260808 20:13:08.262549 11530 raft_consensus.cc:2454] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb8edc2375f449ac806aebab2d5fd6a8 in term 4.
I20260808 20:13:08.263062 11754 consensus_queue.cc:237] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } } peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } }
I20260808 20:13:08.269673 11160 catalog_manager.cc:5495] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 reported cstate change: term changed from 3 to 4, leader changed from 124d7af2a136423c9165edf5e240ec12 (127.10.24.3) to bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1). New cstate: current_term: 4 leader_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:08.319434 11530 raft_consensus.cc:1261] T f2719138be134d1382e8c595a0275d7f P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Refusing update from remote peer bb8edc2375f449ac806aebab2d5fd6a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:08.320459 11754 consensus_queue.cc:1035] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:08.326743 11417 raft_consensus.cc:1261] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98 [term 4 FOLLOWER]: Refusing update from remote peer bb8edc2375f449ac806aebab2d5fd6a8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 20:13:08.328306 11770 consensus_queue.cc:1035] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 20:13:08.333590 11418 raft_consensus.cc:1261] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Refusing update from remote peer bb8edc2375f449ac806aebab2d5fd6a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:08.334913 11770 consensus_queue.cc:1035] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:08.342437 11530 raft_consensus.cc:1261] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 4 FOLLOWER]: Refusing update from remote peer bb8edc2375f449ac806aebab2d5fd6a8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 20:13:08.353240 11786 consensus_queue.cc:1035] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 20:13:08.430259 11530 raft_consensus.cc:1261] T f616636ffca943a7a1731ee069940307 P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Refusing update from remote peer bb8edc2375f449ac806aebab2d5fd6a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:08.431397 11785 consensus_queue.cc:1035] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:08.439564 11418 raft_consensus.cc:1261] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Refusing update from remote peer bb8edc2375f449ac806aebab2d5fd6a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:08.440727 11770 consensus_queue.cc:1035] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:08.571192 11530 raft_consensus.cc:1261] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 3 FOLLOWER]: Refusing update from remote peer bb8edc2375f449ac806aebab2d5fd6a8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:13:08.572216 11770 consensus_queue.cc:1035] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:08.580507 11418 raft_consensus.cc:1261] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 3 FOLLOWER]: Refusing update from remote peer bb8edc2375f449ac806aebab2d5fd6a8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:13:08.581679 11758 consensus_queue.cc:1035] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:08.658070 11530 raft_consensus.cc:1261] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [term 4 FOLLOWER]: Refusing update from remote peer bb8edc2375f449ac806aebab2d5fd6a8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 20:13:08.659155 11758 consensus_queue.cc:1035] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:08.666944 11418 raft_consensus.cc:1261] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 4 FOLLOWER]: Refusing update from remote peer bb8edc2375f449ac806aebab2d5fd6a8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 20:13:08.668241 11785 consensus_queue.cc:1035] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:10.900094 10336 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20260808 20:13:10.900640 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:10.907083 11311 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "261b728800574016968c1083c414e27f"
dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
mode: GRACEFUL
new_leader_uuid: "2fa933dd5405492aa709133798d4ba98"
from {username='slave'} at 127.0.0.1:39632
I20260808 20:13:10.907508 11311 raft_consensus.cc:595] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Received request to transfer leadership to TS 2fa933dd5405492aa709133798d4ba98
I20260808 20:13:10.909538 11311 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "304a2461fac34933aa3a07f54b875889"
dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
mode: GRACEFUL
new_leader_uuid: "124d7af2a136423c9165edf5e240ec12"
from {username='slave'} at 127.0.0.1:39632
I20260808 20:13:10.909951 11311 raft_consensus.cc:595] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 LEADER]: Received request to transfer leadership to TS 124d7af2a136423c9165edf5e240ec12
I20260808 20:13:10.911814 11311 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "941577432a4440b981675cc78e482325"
dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
mode: GRACEFUL
new_leader_uuid: "2fa933dd5405492aa709133798d4ba98"
from {username='slave'} at 127.0.0.1:39632
I20260808 20:13:10.912241 11311 raft_consensus.cc:595] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Received request to transfer leadership to TS 2fa933dd5405492aa709133798d4ba98
I20260808 20:13:10.913990 11311 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d6a5465239d74c75bfae9807d91bccba"
dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
mode: GRACEFUL
new_leader_uuid: "124d7af2a136423c9165edf5e240ec12"
from {username='slave'} at 127.0.0.1:39632
I20260808 20:13:10.914346 11311 raft_consensus.cc:595] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Received request to transfer leadership to TS 124d7af2a136423c9165edf5e240ec12
I20260808 20:13:10.915103 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260808 20:13:10.915463 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:10.942382 11798 raft_consensus.cc:979] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8: : Instructing follower 124d7af2a136423c9165edf5e240ec12 to start an election
I20260808 20:13:10.942752 11758 raft_consensus.cc:1067] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Signalling peer 124d7af2a136423c9165edf5e240ec12 to start an election
I20260808 20:13:10.943912 11530 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d6a5465239d74c75bfae9807d91bccba"
dest_uuid: "124d7af2a136423c9165edf5e240ec12"
from {username='slave'} at 127.0.0.1:52856
I20260808 20:13:10.944290 11530 raft_consensus.cc:482] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:10.944526 11530 raft_consensus.cc:3037] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:10.948455 11530 raft_consensus.cc:504] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:10.950169 11530 leader_election.cc:288] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 3 election: Requested vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075)
I20260808 20:13:10.950809 11311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a5465239d74c75bfae9807d91bccba" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
I20260808 20:13:10.951069 11417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a5465239d74c75bfae9807d91bccba" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2fa933dd5405492aa709133798d4ba98"
I20260808 20:13:10.951419 11311 raft_consensus.cc:3032] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:13:10.951555 11417 raft_consensus.cc:3037] T d6a5465239d74c75bfae9807d91bccba P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:10.951802 11311 raft_consensus.cc:729] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Running, Role: LEADER
I20260808 20:13:10.952642 11311 consensus_queue.cc:260] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:10.953758 11311 raft_consensus.cc:3037] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:10.956739 11417 raft_consensus.cc:2454] T d6a5465239d74c75bfae9807d91bccba P 2fa933dd5405492aa709133798d4ba98 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 3.
I20260808 20:13:10.957705 11475 leader_election.cc:302] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, 2fa933dd5405492aa709133798d4ba98; no voters:
I20260808 20:13:10.958339 11768 raft_consensus.cc:2781] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:13:10.959365 11768 raft_consensus.cc:686] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 3 LEADER]: Becoming Leader. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Running, Role: LEADER
I20260808 20:13:10.959609 11311 raft_consensus.cc:2454] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 3.
I20260808 20:13:10.960243 11768 consensus_queue.cc:237] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:10.973696 11160 catalog_manager.cc:5495] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 reported cstate change: term changed from 2 to 3, leader changed from bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1) to 124d7af2a136423c9165edf5e240ec12 (127.10.24.3). New cstate: current_term: 3 leader_uuid: "124d7af2a136423c9165edf5e240ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: HEALTHY } } }
I20260808 20:13:10.981555 11758 raft_consensus.cc:979] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8: : Instructing follower 2fa933dd5405492aa709133798d4ba98 to start an election
I20260808 20:13:10.982057 11798 raft_consensus.cc:1067] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Signalling peer 2fa933dd5405492aa709133798d4ba98 to start an election
I20260808 20:13:10.983544 11417 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "941577432a4440b981675cc78e482325"
dest_uuid: "2fa933dd5405492aa709133798d4ba98"
from {username='slave'} at 127.0.0.1:47810
I20260808 20:13:10.984009 11417 raft_consensus.cc:482] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:10.984313 11417 raft_consensus.cc:3037] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:10.990046 11417 raft_consensus.cc:504] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:10.991657 11417 leader_election.cc:288] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 3 election: Requested vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783)
I20260808 20:13:10.992513 11530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941577432a4440b981675cc78e482325" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "124d7af2a136423c9165edf5e240ec12"
I20260808 20:13:10.993094 11530 raft_consensus.cc:3037] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:10.993207 11311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941577432a4440b981675cc78e482325" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
I20260808 20:13:10.994132 11311 raft_consensus.cc:3032] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:13:10.994414 11311 raft_consensus.cc:729] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Running, Role: LEADER
I20260808 20:13:10.995086 11311 consensus_queue.cc:260] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:10.996258 11311 raft_consensus.cc:3037] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:10.997911 11530 raft_consensus.cc:2454] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fa933dd5405492aa709133798d4ba98 in term 3.
I20260808 20:13:10.999117 11370 leader_election.cc:302] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, 2fa933dd5405492aa709133798d4ba98; no voters:
I20260808 20:13:10.999878 11712 raft_consensus.cc:2781] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:13:11.001156 11712 raft_consensus.cc:686] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98 [term 3 LEADER]: Becoming Leader. State: Replica: 2fa933dd5405492aa709133798d4ba98, State: Running, Role: LEADER
I20260808 20:13:11.002123 11311 raft_consensus.cc:2454] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fa933dd5405492aa709133798d4ba98 in term 3.
I20260808 20:13:11.002028 11712 consensus_queue.cc:237] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:11.008368 11160 catalog_manager.cc:5495] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98 reported cstate change: term changed from 2 to 3, leader changed from bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1) to 2fa933dd5405492aa709133798d4ba98 (127.10.24.2). New cstate: current_term: 3 leader_uuid: "2fa933dd5405492aa709133798d4ba98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:11.107666 11798 raft_consensus.cc:979] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8: : Instructing follower 124d7af2a136423c9165edf5e240ec12 to start an election
I20260808 20:13:11.108060 11758 raft_consensus.cc:1067] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 LEADER]: Signalling peer 124d7af2a136423c9165edf5e240ec12 to start an election
I20260808 20:13:11.109272 11530 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "304a2461fac34933aa3a07f54b875889"
dest_uuid: "124d7af2a136423c9165edf5e240ec12"
from {username='slave'} at 127.0.0.1:52856
I20260808 20:13:11.109812 11530 raft_consensus.cc:482] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:11.110154 11530 raft_consensus.cc:3037] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:13:11.114321 11530 raft_consensus.cc:504] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:11.116034 11530 leader_election.cc:288] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 4 election: Requested vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 2fa933dd5405492aa709133798d4ba98 (127.10.24.2:33075)
I20260808 20:13:11.116297 11311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304a2461fac34933aa3a07f54b875889" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
I20260808 20:13:11.116547 11417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304a2461fac34933aa3a07f54b875889" candidate_uuid: "124d7af2a136423c9165edf5e240ec12" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2fa933dd5405492aa709133798d4ba98"
I20260808 20:13:11.116840 11311 raft_consensus.cc:3032] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:13:11.117043 11417 raft_consensus.cc:3037] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:13:11.117147 11311 raft_consensus.cc:729] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Running, Role: LEADER
I20260808 20:13:11.117815 11311 consensus_queue.cc:260] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:11.118891 11311 raft_consensus.cc:3037] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:13:11.122717 11417 raft_consensus.cc:2454] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 4.
I20260808 20:13:11.123652 11475 leader_election.cc:302] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, 2fa933dd5405492aa709133798d4ba98; no voters:
I20260808 20:13:11.124173 11311 raft_consensus.cc:2454] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124d7af2a136423c9165edf5e240ec12 in term 4.
I20260808 20:13:11.124279 11768 raft_consensus.cc:2781] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:13:11.124743 11768 raft_consensus.cc:686] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 4 LEADER]: Becoming Leader. State: Replica: 124d7af2a136423c9165edf5e240ec12, State: Running, Role: LEADER
I20260808 20:13:11.125932 11768 consensus_queue.cc:237] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:11.131881 11160 catalog_manager.cc:5495] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 reported cstate change: term changed from 3 to 4, leader changed from bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1) to 124d7af2a136423c9165edf5e240ec12 (127.10.24.3). New cstate: current_term: 4 leader_uuid: "124d7af2a136423c9165edf5e240ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: HEALTHY } } }
I20260808 20:13:11.315557 11798 raft_consensus.cc:979] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8: : Instructing follower 2fa933dd5405492aa709133798d4ba98 to start an election
I20260808 20:13:11.315963 11758 raft_consensus.cc:1067] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Signalling peer 2fa933dd5405492aa709133798d4ba98 to start an election
I20260808 20:13:11.317233 11417 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "261b728800574016968c1083c414e27f"
dest_uuid: "2fa933dd5405492aa709133798d4ba98"
from {username='slave'} at 127.0.0.1:47810
I20260808 20:13:11.317672 11417 raft_consensus.cc:482] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:11.317920 11417 raft_consensus.cc:3037] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:11.321956 11417 raft_consensus.cc:504] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:11.323290 11417 leader_election.cc:288] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 3 election: Requested vote from peers bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959), 124d7af2a136423c9165edf5e240ec12 (127.10.24.3:39783)
I20260808 20:13:11.323928 11311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "261b728800574016968c1083c414e27f" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bb8edc2375f449ac806aebab2d5fd6a8"
I20260808 20:13:11.324301 11530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "261b728800574016968c1083c414e27f" candidate_uuid: "2fa933dd5405492aa709133798d4ba98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "124d7af2a136423c9165edf5e240ec12"
I20260808 20:13:11.324479 11311 raft_consensus.cc:3032] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:13:11.324775 11311 raft_consensus.cc:729] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb8edc2375f449ac806aebab2d5fd6a8, State: Running, Role: LEADER
I20260808 20:13:11.324807 11530 raft_consensus.cc:3037] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:11.325578 11311 consensus_queue.cc:260] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:11.326661 11311 raft_consensus.cc:3037] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:11.330580 11530 raft_consensus.cc:2454] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fa933dd5405492aa709133798d4ba98 in term 3.
I20260808 20:13:11.331508 11370 leader_election.cc:302] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d7af2a136423c9165edf5e240ec12, 2fa933dd5405492aa709133798d4ba98; no voters:
I20260808 20:13:11.332019 11311 raft_consensus.cc:2454] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fa933dd5405492aa709133798d4ba98 in term 3.
I20260808 20:13:11.332111 11712 raft_consensus.cc:2781] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:13:11.332626 11712 raft_consensus.cc:686] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [term 3 LEADER]: Becoming Leader. State: Replica: 2fa933dd5405492aa709133798d4ba98, State: Running, Role: LEADER
I20260808 20:13:11.333390 11712 consensus_queue.cc:237] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } }
I20260808 20:13:11.338961 11160 catalog_manager.cc:5495] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 reported cstate change: term changed from 2 to 3, leader changed from bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1) to 2fa933dd5405492aa709133798d4ba98 (127.10.24.2). New cstate: current_term: 3 leader_uuid: "2fa933dd5405492aa709133798d4ba98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:11.356665 11311 raft_consensus.cc:1261] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 FOLLOWER]: Refusing update from remote peer 124d7af2a136423c9165edf5e240ec12: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:13:11.357642 11768 consensus_queue.cc:1035] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:11.368734 11417 raft_consensus.cc:1261] T d6a5465239d74c75bfae9807d91bccba P 2fa933dd5405492aa709133798d4ba98 [term 3 FOLLOWER]: Refusing update from remote peer 124d7af2a136423c9165edf5e240ec12: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:13:11.369954 11814 consensus_queue.cc:1035] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:11.397331 11530 raft_consensus.cc:1261] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [term 3 FOLLOWER]: Refusing update from remote peer 2fa933dd5405492aa709133798d4ba98: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:13:11.398268 11712 consensus_queue.cc:1035] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:11.405434 11311 raft_consensus.cc:1261] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 FOLLOWER]: Refusing update from remote peer 2fa933dd5405492aa709133798d4ba98: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:13:11.406733 11712 consensus_queue.cc:1035] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260808 20:13:11.516080 11341 debug-util.cc:398] Leaking SignalData structure 0x7b08001b5fa0 after lost signal to thread 10340
W20260808 20:13:11.516808 11341 debug-util.cc:398] Leaking SignalData structure 0x7b0800110380 after lost signal to thread 11206
W20260808 20:13:11.517602 11341 debug-util.cc:398] Leaking SignalData structure 0x7b08000a9100 after lost signal to thread 11345
W20260808 20:13:11.518302 11341 debug-util.cc:398] Leaking SignalData structure 0x7b080010bf00 after lost signal to thread 11454
W20260808 20:13:11.518962 11341 debug-util.cc:398] Leaking SignalData structure 0x7b08001c3fc0 after lost signal to thread 11570
I20260808 20:13:11.585376 11417 raft_consensus.cc:1261] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 4 FOLLOWER]: Refusing update from remote peer 124d7af2a136423c9165edf5e240ec12: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 20:13:11.586534 11814 consensus_queue.cc:1035] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa933dd5405492aa709133798d4ba98" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:11.594981 11311 raft_consensus.cc:1261] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 4 FOLLOWER]: Refusing update from remote peer 124d7af2a136423c9165edf5e240ec12: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 20:13:11.596277 11799 consensus_queue.cc:1035] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:11.770583 11530 raft_consensus.cc:1261] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [term 3 FOLLOWER]: Refusing update from remote peer 2fa933dd5405492aa709133798d4ba98: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:13:11.771831 11712 consensus_queue.cc:1035] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124d7af2a136423c9165edf5e240ec12" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:11.779898 11311 raft_consensus.cc:1261] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 FOLLOWER]: Refusing update from remote peer 2fa933dd5405492aa709133798d4ba98: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:13:11.781533 11712 consensus_queue.cc:1035] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8edc2375f449ac806aebab2d5fd6a8" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:11.916706 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:11.926266 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:13:11.926630 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:12.927395 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:12.933135 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:13:12.933512 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:13.934203 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:13.939134 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:13:13.939504 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:14.940240 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:14.945382 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:13:14.945778 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:15.946513 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:15.951582 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:13:15.951963 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:16.952713 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:16.957963 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:13:16.958343 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:17.959065 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:17.964282 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:13:17.964649 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:18.965487 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:18.971293 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:13:18.971813 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:19.972657 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:19.977669 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 20:13:19.978082 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:21.004447 10336 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20260808 20:13:21.004966 10336 tablet_server.cc:178] TabletServer@127.10.24.1:0 shutting down...
I20260808 20:13:21.039861 10336 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260808 20:13:21.040716 10336 tablet_replica.cc:330] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8: stopping tablet replica
I20260808 20:13:21.041589 10336 raft_consensus.cc:2229] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:21.042201 10336 raft_consensus.cc:2258] T 941577432a4440b981675cc78e482325 P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.044181 10336 tablet_replica.cc:330] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8: stopping tablet replica
I20260808 20:13:21.044865 10336 raft_consensus.cc:2229] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Raft consensus shutting down.
I20260808 20:13:21.045816 10336 raft_consensus.cc:2258] T f2719138be134d1382e8c595a0275d7f P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.048044 10336 tablet_replica.cc:330] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8: stopping tablet replica
I20260808 20:13:21.048728 10336 raft_consensus.cc:2229] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:21.049292 10336 raft_consensus.cc:2258] T 304a2461fac34933aa3a07f54b875889 P bb8edc2375f449ac806aebab2d5fd6a8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.051491 10336 tablet_replica.cc:330] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8: stopping tablet replica
I20260808 20:13:21.052294 10336 raft_consensus.cc:2229] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [term 4 LEADER]: Raft consensus shutting down.
W20260808 20:13:21.052831 11472 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 4 similar messages]
I20260808 20:13:21.053503 10336 raft_consensus.cc:2258] T 44506b0c529c45efbee5ac3003ef6f42 P bb8edc2375f449ac806aebab2d5fd6a8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.055418 10336 tablet_replica.cc:330] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8: stopping tablet replica
I20260808 20:13:21.055868 10336 raft_consensus.cc:2229] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:21.056365 10336 raft_consensus.cc:2258] T d6a5465239d74c75bfae9807d91bccba P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.058579 10336 tablet_replica.cc:330] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8: stopping tablet replica
I20260808 20:13:21.059039 10336 raft_consensus.cc:2229] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 LEADER]: Raft consensus shutting down.
I20260808 20:13:21.059762 10336 raft_consensus.cc:2258] T f616636ffca943a7a1731ee069940307 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.061960 10336 tablet_replica.cc:330] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8: stopping tablet replica
I20260808 20:13:21.062654 10336 raft_consensus.cc:2229] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:21.063246 10336 raft_consensus.cc:2258] T 089736c28b834bd5a121f818bcb364d7 P bb8edc2375f449ac806aebab2d5fd6a8 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260808 20:13:21.063550 11472 consensus_peers.cc:487] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 -> Peer bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959): Couldn't send request to peer bb8edc2375f449ac806aebab2d5fd6a8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:21.065006 10336 tablet_replica.cc:330] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8: stopping tablet replica
I20260808 20:13:21.065455 10336 raft_consensus.cc:2229] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:21.065814 10336 raft_consensus.cc:2258] T 261b728800574016968c1083c414e27f P bb8edc2375f449ac806aebab2d5fd6a8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.067987 10336 tablet_replica.cc:330] T 07fe59ac00b24efb8271f8e86a33441b P bb8edc2375f449ac806aebab2d5fd6a8: stopping tablet replica
I20260808 20:13:21.068598 10336 raft_consensus.cc:2229] T 07fe59ac00b24efb8271f8e86a33441b P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:21.069161 10336 raft_consensus.cc:2258] T 07fe59ac00b24efb8271f8e86a33441b P bb8edc2375f449ac806aebab2d5fd6a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.071453 10336 tablet_replica.cc:330] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8: stopping tablet replica
I20260808 20:13:21.072016 10336 raft_consensus.cc:2229] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 4 LEADER]: Raft consensus shutting down.
I20260808 20:13:21.072914 10336 raft_consensus.cc:2258] T 5e4e13f27b9d4abbbda7599371e4ad48 P bb8edc2375f449ac806aebab2d5fd6a8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.119158 10336 tablet_server.cc:194] TabletServer@127.10.24.1:0 shutdown complete.
I20260808 20:13:21.138218 10336 tablet_server.cc:178] TabletServer@127.10.24.2:0 shutting down...
W20260808 20:13:21.141832 11365 consensus_peers.cc:487] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 -> Peer bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959): Couldn't send request to peer bb8edc2375f449ac806aebab2d5fd6a8. Status: Network error: Client connection negotiation failed: client connection to 127.10.24.1:40959: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 20:13:21.161641 11472 consensus_peers.cc:487] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 -> Peer bb8edc2375f449ac806aebab2d5fd6a8 (127.10.24.1:40959): Couldn't send request to peer bb8edc2375f449ac806aebab2d5fd6a8. Status: Network error: Client connection negotiation failed: client connection to 127.10.24.1:40959: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:21.166697 10336 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260808 20:13:21.167307 10336 tablet_replica.cc:330] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98: stopping tablet replica
I20260808 20:13:21.167855 10336 raft_consensus.cc:2229] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98 [term 3 LEADER]: Raft consensus shutting down.
I20260808 20:13:21.168555 10336 raft_consensus.cc:2258] T 941577432a4440b981675cc78e482325 P 2fa933dd5405492aa709133798d4ba98 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.170261 10336 tablet_replica.cc:330] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98: stopping tablet replica
I20260808 20:13:21.170749 10336 raft_consensus.cc:2229] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98 [term 2 LEADER]: Raft consensus shutting down.
I20260808 20:13:21.171468 10336 raft_consensus.cc:2258] T 089736c28b834bd5a121f818bcb364d7 P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.173292 10336 tablet_replica.cc:330] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98: stopping tablet replica
I20260808 20:13:21.173750 10336 raft_consensus.cc:2229] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:21.174145 10336 raft_consensus.cc:2258] T 304a2461fac34933aa3a07f54b875889 P 2fa933dd5405492aa709133798d4ba98 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.175607 10336 tablet_replica.cc:330] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98: stopping tablet replica
I20260808 20:13:21.176050 10336 raft_consensus.cc:2229] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:21.176426 10336 raft_consensus.cc:2258] T f2719138be134d1382e8c595a0275d7f P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.177944 10336 tablet_replica.cc:330] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98: stopping tablet replica
I20260808 20:13:21.178382 10336 raft_consensus.cc:2229] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [term 3 LEADER]: Raft consensus shutting down.
I20260808 20:13:21.179051 10336 raft_consensus.cc:2258] T 261b728800574016968c1083c414e27f P 2fa933dd5405492aa709133798d4ba98 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.180554 10336 tablet_replica.cc:330] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98: stopping tablet replica
I20260808 20:13:21.181058 10336 raft_consensus.cc:2229] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:21.181485 10336 raft_consensus.cc:2258] T 5e4e13f27b9d4abbbda7599371e4ad48 P 2fa933dd5405492aa709133798d4ba98 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.183295 10336 tablet_replica.cc:330] T 07fe59ac00b24efb8271f8e86a33441b P 2fa933dd5405492aa709133798d4ba98: stopping tablet replica
I20260808 20:13:21.183961 10336 raft_consensus.cc:2229] T 07fe59ac00b24efb8271f8e86a33441b P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:21.184409 10336 raft_consensus.cc:2258] T 07fe59ac00b24efb8271f8e86a33441b P 2fa933dd5405492aa709133798d4ba98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.186501 10336 tablet_replica.cc:330] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98: stopping tablet replica
I20260808 20:13:21.187115 10336 raft_consensus.cc:2229] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:21.187486 10336 raft_consensus.cc:2258] T f616636ffca943a7a1731ee069940307 P 2fa933dd5405492aa709133798d4ba98 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.189085 10336 tablet_replica.cc:330] T d6a5465239d74c75bfae9807d91bccba P 2fa933dd5405492aa709133798d4ba98: stopping tablet replica
I20260808 20:13:21.189466 10336 raft_consensus.cc:2229] T d6a5465239d74c75bfae9807d91bccba P 2fa933dd5405492aa709133798d4ba98 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:21.189800 10336 raft_consensus.cc:2258] T d6a5465239d74c75bfae9807d91bccba P 2fa933dd5405492aa709133798d4ba98 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.191184 10336 tablet_replica.cc:330] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98: stopping tablet replica
I20260808 20:13:21.191641 10336 raft_consensus.cc:2229] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:21.191977 10336 raft_consensus.cc:2258] T 44506b0c529c45efbee5ac3003ef6f42 P 2fa933dd5405492aa709133798d4ba98 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.220419 10336 tablet_server.cc:194] TabletServer@127.10.24.2:0 shutdown complete.
I20260808 20:13:21.238647 10336 tablet_server.cc:178] TabletServer@127.10.24.3:0 shutting down...
I20260808 20:13:21.260947 10336 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260808 20:13:21.261597 10336 tablet_replica.cc:330] T f616636ffca943a7a1731ee069940307 P 124d7af2a136423c9165edf5e240ec12: stopping tablet replica
I20260808 20:13:21.262342 10336 raft_consensus.cc:2229] T f616636ffca943a7a1731ee069940307 P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:21.262919 10336 raft_consensus.cc:2258] T f616636ffca943a7a1731ee069940307 P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.264804 10336 tablet_replica.cc:330] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12: stopping tablet replica
I20260808 20:13:21.265296 10336 raft_consensus.cc:2229] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12 [term 1 LEADER]: Raft consensus shutting down.
I20260808 20:13:21.265990 10336 raft_consensus.cc:2258] T 07fe59ac00b24efb8271f8e86a33441b P 124d7af2a136423c9165edf5e240ec12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.267551 10336 tablet_replica.cc:330] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12: stopping tablet replica
I20260808 20:13:21.268002 10336 raft_consensus.cc:2229] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:21.268337 10336 raft_consensus.cc:2258] T 5e4e13f27b9d4abbbda7599371e4ad48 P 124d7af2a136423c9165edf5e240ec12 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.269912 10336 tablet_replica.cc:330] T f2719138be134d1382e8c595a0275d7f P 124d7af2a136423c9165edf5e240ec12: stopping tablet replica
I20260808 20:13:21.270342 10336 raft_consensus.cc:2229] T f2719138be134d1382e8c595a0275d7f P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:21.270686 10336 raft_consensus.cc:2258] T f2719138be134d1382e8c595a0275d7f P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.272133 10336 tablet_replica.cc:330] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12: stopping tablet replica
I20260808 20:13:21.272547 10336 raft_consensus.cc:2229] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 4 LEADER]: Raft consensus shutting down.
I20260808 20:13:21.273239 10336 raft_consensus.cc:2258] T 304a2461fac34933aa3a07f54b875889 P 124d7af2a136423c9165edf5e240ec12 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.274744 10336 tablet_replica.cc:330] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12: stopping tablet replica
I20260808 20:13:21.275152 10336 raft_consensus.cc:2229] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:21.275496 10336 raft_consensus.cc:2258] T 089736c28b834bd5a121f818bcb364d7 P 124d7af2a136423c9165edf5e240ec12 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.276880 10336 tablet_replica.cc:330] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12: stopping tablet replica
I20260808 20:13:21.277331 10336 raft_consensus.cc:2229] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:21.277694 10336 raft_consensus.cc:2258] T 44506b0c529c45efbee5ac3003ef6f42 P 124d7af2a136423c9165edf5e240ec12 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.279062 10336 tablet_replica.cc:330] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12: stopping tablet replica
I20260808 20:13:21.279449 10336 raft_consensus.cc:2229] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 3 LEADER]: Raft consensus shutting down.
I20260808 20:13:21.280097 10336 raft_consensus.cc:2258] T d6a5465239d74c75bfae9807d91bccba P 124d7af2a136423c9165edf5e240ec12 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.281607 10336 tablet_replica.cc:330] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12: stopping tablet replica
I20260808 20:13:21.281988 10336 raft_consensus.cc:2229] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:21.282363 10336 raft_consensus.cc:2258] T 261b728800574016968c1083c414e27f P 124d7af2a136423c9165edf5e240ec12 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.283707 10336 tablet_replica.cc:330] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12: stopping tablet replica
I20260808 20:13:21.284054 10336 raft_consensus.cc:2229] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:21.284404 10336 raft_consensus.cc:2258] T 941577432a4440b981675cc78e482325 P 124d7af2a136423c9165edf5e240ec12 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.312929 10336 tablet_server.cc:194] TabletServer@127.10.24.3:0 shutdown complete.
I20260808 20:13:21.330174 10336 master.cc:537] Master@127.10.24.62:46075 shutting down...
I20260808 20:13:21.350896 10336 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89 [term 1 LEADER]: Raft consensus shutting down.
I20260808 20:13:21.351425 10336 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:21.351735 10336 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4b6c1038515743ca968bf5357b32af89: stopping tablet replica
I20260808 20:13:21.369926 10336 master.cc:552] Master@127.10.24.62:46075 shutdown complete.
124d7af2a136423c9165edf5e240ec12 1
2fa933dd5405492aa709133798d4ba98 4
bb8edc2375f449ac806aebab2d5fd6a8 5
124d7af2a136423c9165edf5e240ec12 3
2fa933dd5405492aa709133798d4ba98 4
bb8edc2375f449ac806aebab2d5fd6a8 3
124d7af2a136423c9165edf5e240ec12 1
2fa933dd5405492aa709133798d4ba98 1
bb8edc2375f449ac806aebab2d5fd6a8 8
124d7af2a136423c9165edf5e240ec12 3
2fa933dd5405492aa709133798d4ba98 3
bb8edc2375f449ac806aebab2d5fd6a8 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28324 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260808 20:13:21.409567 10336 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.10.24.62:36417
I20260808 20:13:21.410614 10336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 20:13:21.415951 11851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 20:13:21.415987 11852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 20:13:21.417390 11857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 20:13:21.418303 10336 server_base.cc:1034] running on GCE node
I20260808 20:13:21.419298 10336 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 20:13:21.419469 10336 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 20:13:21.419579 10336 hybrid_clock.cc:648] HybridClock initialized: now 1786220001419569 us; error 0 us; skew 500 ppm
I20260808 20:13:21.419999 10336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 20:13:21.422350 10336 webserver.cc:460] Webserver started at http://127.10.24.62:40519/ using document root <none> and password file <none>
I20260808 20:13:21.422752 10336 fs_manager.cc:359] Metadata directory not provided
I20260808 20:13:21.422899 10336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 20:13:21.423132 10336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 20:13:21.424250 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/master-0-root/instance:
uuid: "9da0bf47b60c4880bc83222a7feaf045"
format_stamp: "Formatted at 2026-08-08 20:13:21 on dist-test-slave-ncp9"
I20260808 20:13:21.428480 10336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 20:13:21.431576 11868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 20:13:21.432225 10336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 20:13:21.432468 10336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/master-0-root
uuid: "9da0bf47b60c4880bc83222a7feaf045"
format_stamp: "Formatted at 2026-08-08 20:13:21 on dist-test-slave-ncp9"
I20260808 20:13:21.432724 10336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 20:13:21.450591 10336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 20:13:21.452003 10336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 20:13:21.488528 10336 rpc_server.cc:307] RPC server started. Bound to: 127.10.24.62:36417
I20260808 20:13:21.488649 11946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.24.62:36417 every 8 connection(s)
I20260808 20:13:21.492246 11948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:21.501849 11948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045: Bootstrap starting.
I20260808 20:13:21.506043 11948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:21.509966 11948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045: No bootstrap required, opened a new log
I20260808 20:13:21.511945 11948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da0bf47b60c4880bc83222a7feaf045" member_type: VOTER }
I20260808 20:13:21.512398 11948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:21.512634 11948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da0bf47b60c4880bc83222a7feaf045, State: Initialized, Role: FOLLOWER
I20260808 20:13:21.513195 11948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da0bf47b60c4880bc83222a7feaf045" member_type: VOTER }
I20260808 20:13:21.513613 11948 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 20:13:21.513833 11948 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 20:13:21.514091 11948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:21.517915 11948 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da0bf47b60c4880bc83222a7feaf045" member_type: VOTER }
I20260808 20:13:21.518368 11948 leader_election.cc:302] T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9da0bf47b60c4880bc83222a7feaf045; no voters:
I20260808 20:13:21.519397 11948 leader_election.cc:288] T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 20:13:21.519753 11953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:21.520942 11953 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045 [term 1 LEADER]: Becoming Leader. State: Replica: 9da0bf47b60c4880bc83222a7feaf045, State: Running, Role: LEADER
I20260808 20:13:21.521584 11953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da0bf47b60c4880bc83222a7feaf045" member_type: VOTER }
I20260808 20:13:21.522097 11948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 20:13:21.526283 11957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9da0bf47b60c4880bc83222a7feaf045. Latest consensus state: current_term: 1 leader_uuid: "9da0bf47b60c4880bc83222a7feaf045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da0bf47b60c4880bc83222a7feaf045" member_type: VOTER } }
I20260808 20:13:21.526367 11954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9da0bf47b60c4880bc83222a7feaf045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da0bf47b60c4880bc83222a7feaf045" member_type: VOTER } }
I20260808 20:13:21.526871 11957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045 [sys.catalog]: This master's current role is: LEADER
I20260808 20:13:21.526989 11954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045 [sys.catalog]: This master's current role is: LEADER
I20260808 20:13:21.528784 11964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 20:13:21.534801 11964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 20:13:21.535743 10336 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260808 20:13:21.542758 11964 catalog_manager.cc:1347] Generated new cluster ID: 29f5c8c9f8cc49f39e8158192f936543
I20260808 20:13:21.543041 11964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 20:13:21.571669 11964 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 20:13:21.573304 11964 catalog_manager.cc:1504] Loading token signing keys...
I20260808 20:13:21.588091 11964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045: Generated new TSK 0
I20260808 20:13:21.588701 11964 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 20:13:21.601934 10336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 20:13:21.607395 11981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 20:13:21.608325 11982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 20:13:21.610038 11985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 20:13:21.610745 10336 server_base.cc:1034] running on GCE node
I20260808 20:13:21.611681 10336 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 20:13:21.611857 10336 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 20:13:21.611996 10336 hybrid_clock.cc:648] HybridClock initialized: now 1786220001611983 us; error 0 us; skew 500 ppm
I20260808 20:13:21.612474 10336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 20:13:21.614784 10336 webserver.cc:460] Webserver started at http://127.10.24.1:33309/ using document root <none> and password file <none>
I20260808 20:13:21.615219 10336 fs_manager.cc:359] Metadata directory not provided
I20260808 20:13:21.615386 10336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 20:13:21.615631 10336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 20:13:21.616647 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-0-root/instance:
uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
format_stamp: "Formatted at 2026-08-08 20:13:21 on dist-test-slave-ncp9"
I20260808 20:13:21.620820 10336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 20:13:21.623998 11993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 20:13:21.624639 10336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 20:13:21.625208 10336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-0-root
uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
format_stamp: "Formatted at 2026-08-08 20:13:21 on dist-test-slave-ncp9"
I20260808 20:13:21.625473 10336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 20:13:21.645311 10336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 20:13:21.646448 10336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 20:13:21.647917 10336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 20:13:21.650084 10336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 20:13:21.650267 10336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 20:13:21.650477 10336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 20:13:21.650637 10336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 20:13:21.688884 10336 rpc_server.cc:307] RPC server started. Bound to: 127.10.24.1:40747
I20260808 20:13:21.689003 12083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.24.1:40747 every 8 connection(s)
I20260808 20:13:21.693486 10336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 20:13:21.699998 12089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 20:13:21.701923 12090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 20:13:21.702818 12094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 20:13:21.704418 10336 server_base.cc:1034] running on GCE node
I20260808 20:13:21.705615 10336 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 20:13:21.705856 10336 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 20:13:21.706029 10336 hybrid_clock.cc:648] HybridClock initialized: now 1786220001706014 us; error 0 us; skew 500 ppm
I20260808 20:13:21.706547 10336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 20:13:21.707023 12084 heartbeater.cc:347] Connected to a master server at 127.10.24.62:36417
I20260808 20:13:21.707330 12084 heartbeater.cc:464] Registering TS with master...
I20260808 20:13:21.707958 12084 heartbeater.cc:511] Master 127.10.24.62:36417 requested a full tablet report, sending...
I20260808 20:13:21.709189 10336 webserver.cc:460] Webserver started at http://127.10.24.2:35271/ using document root <none> and password file <none>
I20260808 20:13:21.709738 10336 fs_manager.cc:359] Metadata directory not provided
I20260808 20:13:21.709919 10336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 20:13:21.710011 11897 ts_manager.cc:194] Registered new tserver with Master: 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:13:21.710155 10336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 20:13:21.711345 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-1-root/instance:
uuid: "78b9263889cd4f1582578a3aa4f644d9"
format_stamp: "Formatted at 2026-08-08 20:13:21 on dist-test-slave-ncp9"
I20260808 20:13:21.711938 11897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57474
I20260808 20:13:21.715637 10336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260808 20:13:21.718673 12099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 20:13:21.719336 10336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 20:13:21.719621 10336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-1-root
uuid: "78b9263889cd4f1582578a3aa4f644d9"
format_stamp: "Formatted at 2026-08-08 20:13:21 on dist-test-slave-ncp9"
I20260808 20:13:21.719871 10336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 20:13:21.741246 10336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 20:13:21.742342 10336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 20:13:21.743697 10336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 20:13:21.745790 10336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 20:13:21.745970 10336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 20:13:21.746189 10336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 20:13:21.746330 10336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 20:13:21.783070 10336 rpc_server.cc:307] RPC server started. Bound to: 127.10.24.2:35139
I20260808 20:13:21.783183 12185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.24.2:35139 every 8 connection(s)
I20260808 20:13:21.787317 10336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 20:13:21.794878 12192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 20:13:21.795030 12191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 20:13:21.797559 12194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 20:13:21.798488 10336 server_base.cc:1034] running on GCE node
I20260808 20:13:21.799381 10336 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 20:13:21.799608 10336 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 20:13:21.799800 10336 hybrid_clock.cc:648] HybridClock initialized: now 1786220001799784 us; error 0 us; skew 500 ppm
I20260808 20:13:21.800437 10336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 20:13:21.801859 12186 heartbeater.cc:347] Connected to a master server at 127.10.24.62:36417
I20260808 20:13:21.802198 12186 heartbeater.cc:464] Registering TS with master...
I20260808 20:13:21.802887 12186 heartbeater.cc:511] Master 127.10.24.62:36417 requested a full tablet report, sending...
I20260808 20:13:21.803061 10336 webserver.cc:460] Webserver started at http://127.10.24.3:44397/ using document root <none> and password file <none>
I20260808 20:13:21.803587 10336 fs_manager.cc:359] Metadata directory not provided
I20260808 20:13:21.803813 10336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 20:13:21.804114 10336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 20:13:21.804832 11897 ts_manager.cc:194] Registered new tserver with Master: 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:21.805567 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-2-root/instance:
uuid: "b9d94f35ea15421aa1d721201c782d8a"
format_stamp: "Formatted at 2026-08-08 20:13:21 on dist-test-slave-ncp9"
I20260808 20:13:21.806869 11897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57488
I20260808 20:13:21.810397 10336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 20:13:21.813414 12202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 20:13:21.814071 10336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260808 20:13:21.814316 10336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-2-root
uuid: "b9d94f35ea15421aa1d721201c782d8a"
format_stamp: "Formatted at 2026-08-08 20:13:21 on dist-test-slave-ncp9"
I20260808 20:13:21.814576 10336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 20:13:21.836655 10336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 20:13:21.837736 10336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 20:13:21.839067 10336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 20:13:21.841111 10336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 20:13:21.841289 10336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 20:13:21.841493 10336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 20:13:21.841640 10336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 20:13:21.878811 10336 rpc_server.cc:307] RPC server started. Bound to: 127.10.24.3:38391
I20260808 20:13:21.878913 12287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.24.3:38391 every 8 connection(s)
I20260808 20:13:21.892828 12288 heartbeater.cc:347] Connected to a master server at 127.10.24.62:36417
I20260808 20:13:21.893195 12288 heartbeater.cc:464] Registering TS with master...
I20260808 20:13:21.893843 12288 heartbeater.cc:511] Master 127.10.24.62:36417 requested a full tablet report, sending...
I20260808 20:13:21.914999 10336 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.033539826s
I20260808 20:13:21.915063 11897 ts_manager.cc:194] Registered new tserver with Master: b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:21.919282 11897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57500
I20260808 20:13:21.923527 10336 test_util.cc:268] Using random seed: -1061908049
I20260808 20:13:21.949560 11897 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57516:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 20:13:21.955859 11897 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 20:13:22.168444 12136 tablet_service.cc:1467] Processing CreateTablet for tablet cb8b79bb1c59483e93af813653a73a31 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260808 20:13:22.170226 12136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb8b79bb1c59483e93af813653a73a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.174124 12134 tablet_service.cc:1467] Processing CreateTablet for tablet fd0cef71281c44f6b3fd2afe876735f8 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260808 20:13:22.175490 12134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0cef71281c44f6b3fd2afe876735f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.178099 12132 tablet_service.cc:1467] Processing CreateTablet for tablet 4e4bf0005d9e43c1a4adce7588e7653f (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260808 20:13:22.179394 12132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e4bf0005d9e43c1a4adce7588e7653f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.182082 12131 tablet_service.cc:1467] Processing CreateTablet for tablet 358aa11611bd4b16a062e6ced419bef8 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260808 20:13:22.183460 12126 tablet_service.cc:1467] Processing CreateTablet for tablet f73cf4b21a754e75b2af936d7402def2 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260808 20:13:22.187948 12034 tablet_service.cc:1467] Processing CreateTablet for tablet fd0cef71281c44f6b3fd2afe876735f8 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260808 20:13:22.188432 12131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 358aa11611bd4b16a062e6ced419bef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.188688 12125 tablet_service.cc:1467] Processing CreateTablet for tablet f4918de0ffe94198a66ac1975ddbc693 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260808 20:13:22.189440 12034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0cef71281c44f6b3fd2afe876735f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.190256 12125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4918de0ffe94198a66ac1975ddbc693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.185776 12127 tablet_service.cc:1467] Processing CreateTablet for tablet 2d977879a1ef46db915d4b277255a08c (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260808 20:13:22.193583 12126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f73cf4b21a754e75b2af936d7402def2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.194155 12033 tablet_service.cc:1467] Processing CreateTablet for tablet 4e4bf0005d9e43c1a4adce7588e7653f (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260808 20:13:22.195087 12124 tablet_service.cc:1467] Processing CreateTablet for tablet 0d91cb12be034dcc9ca3038df3cc85cf (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260808 20:13:22.184617 12128 tablet_service.cc:1467] Processing CreateTablet for tablet 72e59eff611e4cc693407979ebe0857c (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260808 20:13:22.196410 12124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d91cb12be034dcc9ca3038df3cc85cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.197113 12128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e59eff611e4cc693407979ebe0857c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.195477 12033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e4bf0005d9e43c1a4adce7588e7653f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.186273 12035 tablet_service.cc:1467] Processing CreateTablet for tablet cb8b79bb1c59483e93af813653a73a31 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260808 20:13:22.201232 12127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d977879a1ef46db915d4b277255a08c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.204061 12031 tablet_service.cc:1467] Processing CreateTablet for tablet 358aa11611bd4b16a062e6ced419bef8 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260808 20:13:22.205343 12031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 358aa11611bd4b16a062e6ced419bef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.208432 12028 tablet_service.cc:1467] Processing CreateTablet for tablet 2d977879a1ef46db915d4b277255a08c (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260808 20:13:22.209192 12035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb8b79bb1c59483e93af813653a73a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.210638 12030 tablet_service.cc:1467] Processing CreateTablet for tablet 72e59eff611e4cc693407979ebe0857c (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260808 20:13:22.211851 12030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e59eff611e4cc693407979ebe0857c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.213923 12028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d977879a1ef46db915d4b277255a08c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.214553 12024 tablet_service.cc:1467] Processing CreateTablet for tablet f4918de0ffe94198a66ac1975ddbc693 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260808 20:13:22.215829 12024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4918de0ffe94198a66ac1975ddbc693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.223273 12123 tablet_service.cc:1467] Processing CreateTablet for tablet aacf7252c47a48cdb408ea9e49270f80 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260808 20:13:22.208384 12025 tablet_service.cc:1467] Processing CreateTablet for tablet f73cf4b21a754e75b2af936d7402def2 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260808 20:13:22.230154 12025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f73cf4b21a754e75b2af936d7402def2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.232031 12127 tablet_service.cc:1467] Processing CreateTablet for tablet ddb83ac6240244be967995e7771a8b64 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260808 20:13:22.233213 12127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddb83ac6240244be967995e7771a8b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.234400 12023 tablet_service.cc:1467] Processing CreateTablet for tablet 0d91cb12be034dcc9ca3038df3cc85cf (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260808 20:13:22.235673 12023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d91cb12be034dcc9ca3038df3cc85cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.236019 12123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aacf7252c47a48cdb408ea9e49270f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.247630 12127 tablet_service.cc:1467] Processing CreateTablet for tablet a9ef909a266f4100bd54d95062fc7e25 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260808 20:13:22.248855 12127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ef909a266f4100bd54d95062fc7e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.260507 12022 tablet_service.cc:1467] Processing CreateTablet for tablet aacf7252c47a48cdb408ea9e49270f80 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260808 20:13:22.261931 12022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aacf7252c47a48cdb408ea9e49270f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.264349 12134 tablet_service.cc:1467] Processing CreateTablet for tablet eec183c460ad4f1b85ad1407c53f2eab (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260808 20:13:22.265522 12134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eec183c460ad4f1b85ad1407c53f2eab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.268077 12309 tablet_bootstrap.cc:492] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:22.275635 12237 tablet_service.cc:1467] Processing CreateTablet for tablet cb8b79bb1c59483e93af813653a73a31 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260808 20:13:22.277088 12237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb8b79bb1c59483e93af813653a73a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.282568 12235 tablet_service.cc:1467] Processing CreateTablet for tablet fd0cef71281c44f6b3fd2afe876735f8 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260808 20:13:22.283871 12235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0cef71281c44f6b3fd2afe876735f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.290488 12234 tablet_service.cc:1467] Processing CreateTablet for tablet 4e4bf0005d9e43c1a4adce7588e7653f (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260808 20:13:22.290738 12127 tablet_service.cc:1467] Processing CreateTablet for tablet d092815e2e3e40ba8cbcc396cbeeb3b7 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260808 20:13:22.292052 12234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e4bf0005d9e43c1a4adce7588e7653f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.298462 12231 tablet_service.cc:1467] Processing CreateTablet for tablet 358aa11611bd4b16a062e6ced419bef8 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260808 20:13:22.298610 12128 tablet_service.cc:1467] Processing CreateTablet for tablet 4b5f5f3a393f488ca713b7269b5dfa78 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260808 20:13:22.299789 12231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 358aa11611bd4b16a062e6ced419bef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.299814 12128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b5f5f3a393f488ca713b7269b5dfa78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.303454 12229 tablet_service.cc:1467] Processing CreateTablet for tablet 2d977879a1ef46db915d4b277255a08c (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260808 20:13:22.304695 12229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d977879a1ef46db915d4b277255a08c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.306069 12228 tablet_service.cc:1467] Processing CreateTablet for tablet f73cf4b21a754e75b2af936d7402def2 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260808 20:13:22.307719 12224 tablet_service.cc:1467] Processing CreateTablet for tablet 0d91cb12be034dcc9ca3038df3cc85cf (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260808 20:13:22.309437 12223 tablet_service.cc:1467] Processing CreateTablet for tablet aacf7252c47a48cdb408ea9e49270f80 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260808 20:13:22.309986 12136 tablet_service.cc:1467] Processing CreateTablet for tablet 9d6f3bbe5e92475487be203c7c89b73d (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260808 20:13:22.314718 12127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d092815e2e3e40ba8cbcc396cbeeb3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.318133 12230 tablet_service.cc:1467] Processing CreateTablet for tablet 72e59eff611e4cc693407979ebe0857c (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260808 20:13:22.318377 12126 tablet_service.cc:1467] Processing CreateTablet for tablet 0be6c610fe68472b8bcb6232b862dbc1 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260808 20:13:22.320286 12309 tablet_bootstrap.cc:654] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:22.326961 12225 tablet_service.cc:1467] Processing CreateTablet for tablet f4918de0ffe94198a66ac1975ddbc693 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260808 20:13:22.329236 12228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f73cf4b21a754e75b2af936d7402def2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.329957 12225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4918de0ffe94198a66ac1975ddbc693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.332126 12224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d91cb12be034dcc9ca3038df3cc85cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.333416 12230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e59eff611e4cc693407979ebe0857c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.335940 12136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d6f3bbe5e92475487be203c7c89b73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.337062 12035 tablet_service.cc:1467] Processing CreateTablet for tablet eec183c460ad4f1b85ad1407c53f2eab (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260808 20:13:22.337687 12223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aacf7252c47a48cdb408ea9e49270f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.338233 12035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eec183c460ad4f1b85ad1407c53f2eab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.343608 12134 tablet_service.cc:1467] Processing CreateTablet for tablet 5e931b9df2c74729a3771d5f0734f559 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260808 20:13:22.343997 12125 tablet_service.cc:1467] Processing CreateTablet for tablet 7aff1371284542e48cbd467d3ef0ede9 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260808 20:13:22.344810 12134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e931b9df2c74729a3771d5f0734f559. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.358443 12030 tablet_service.cc:1467] Processing CreateTablet for tablet ddb83ac6240244be967995e7771a8b64 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260808 20:13:22.359642 12030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddb83ac6240244be967995e7771a8b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.365208 12033 tablet_service.cc:1467] Processing CreateTablet for tablet a9ef909a266f4100bd54d95062fc7e25 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260808 20:13:22.366405 12033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ef909a266f4100bd54d95062fc7e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.367864 12132 tablet_service.cc:1467] Processing CreateTablet for tablet e5dc2ab0712f4054883178f4b4c50928 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260808 20:13:22.381291 12311 tablet_bootstrap.cc:492] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:22.382699 12125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aff1371284542e48cbd467d3ef0ede9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.384279 12132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5dc2ab0712f4054883178f4b4c50928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.385102 12126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be6c610fe68472b8bcb6232b862dbc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.388490 12031 tablet_service.cc:1467] Processing CreateTablet for tablet d092815e2e3e40ba8cbcc396cbeeb3b7 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260808 20:13:22.389715 12031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d092815e2e3e40ba8cbcc396cbeeb3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.434154 12034 tablet_service.cc:1467] Processing CreateTablet for tablet e5dc2ab0712f4054883178f4b4c50928 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260808 20:13:22.435469 12034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5dc2ab0712f4054883178f4b4c50928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.438871 12028 tablet_service.cc:1467] Processing CreateTablet for tablet 9d6f3bbe5e92475487be203c7c89b73d (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260808 20:13:22.440058 12028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d6f3bbe5e92475487be203c7c89b73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.446280 12023 tablet_service.cc:1467] Processing CreateTablet for tablet 4b5f5f3a393f488ca713b7269b5dfa78 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260808 20:13:22.451828 12131 tablet_service.cc:1467] Processing CreateTablet for tablet 2a41c716cfc94652a2fc6957a1bd2a78 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260808 20:13:22.448086 12313 tablet_bootstrap.cc:492] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:22.454955 12309 tablet_bootstrap.cc:492] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:22.455443 12309 ts_tablet_manager.cc:1397] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.188s user 0.006s sys 0.015s
I20260808 20:13:22.459813 12128 tablet_service.cc:1467] Processing CreateTablet for tablet a83f6daa15a54982865040c0110bdda8 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260808 20:13:22.468570 12235 tablet_service.cc:1467] Processing CreateTablet for tablet 2cd3d6998af44e69a7182125e02f16e2 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260808 20:13:22.471057 12235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cd3d6998af44e69a7182125e02f16e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.475217 12311 tablet_bootstrap.cc:654] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:22.476011 12123 tablet_service.cc:1467] Processing CreateTablet for tablet 4e5943c3d789423691d3fe7099372c59 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260808 20:13:22.481303 12131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a41c716cfc94652a2fc6957a1bd2a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.463104 12025 tablet_service.cc:1467] Processing CreateTablet for tablet 0be6c610fe68472b8bcb6232b862dbc1 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260808 20:13:22.483479 12128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83f6daa15a54982865040c0110bdda8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.486968 12123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e5943c3d789423691d3fe7099372c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.491873 12023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b5f5f3a393f488ca713b7269b5dfa78. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 20:13:22.493258 11875 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 156036us
I20260808 20:13:22.507019 12025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be6c610fe68472b8bcb6232b862dbc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.512622 12313 tablet_bootstrap.cc:654] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
W20260808 20:13:22.513988 11872 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 47286.3us
W20260808 20:13:22.531652 11966 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260808 20:13:22.514671 11873 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 177173us
I20260808 20:13:22.539652 12223 tablet_service.cc:1467] Processing CreateTablet for tablet eec183c460ad4f1b85ad1407c53f2eab (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260808 20:13:22.529870 12124 tablet_service.cc:1467] Processing CreateTablet for tablet a383bcab83014009b5698c63cdac298b (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260808 20:13:22.516378 12024 tablet_service.cc:1467] Processing CreateTablet for tablet 5e931b9df2c74729a3771d5f0734f559 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260808 20:13:22.529249 12309 raft_consensus.cc:348] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } }
I20260808 20:13:22.549036 12309 raft_consensus.cc:374] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:22.549391 12309 raft_consensus.cc:729] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:22.550136 12309 consensus_queue.cc:260] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } }
I20260808 20:13:22.551599 12022 tablet_service.cc:1467] Processing CreateTablet for tablet a83f6daa15a54982865040c0110bdda8 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260808 20:13:22.531670 12127 tablet_service.cc:1467] Processing CreateTablet for tablet 85cecc37ef974ec58b1d5fe18e9801a2 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
W20260808 20:13:22.556380 11875 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 62275.5us
I20260808 20:13:22.557636 12127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85cecc37ef974ec58b1d5fe18e9801a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.552872 12022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83f6daa15a54982865040c0110bdda8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.568011 12132 tablet_service.cc:1467] Processing CreateTablet for tablet 4e393e90875847809e1ef94179c5b4ef (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260808 20:13:22.555920 12031 tablet_service.cc:1467] Processing CreateTablet for tablet 7aff1371284542e48cbd467d3ef0ede9 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260808 20:13:22.583776 12024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e931b9df2c74729a3771d5f0734f559. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.584425 12031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aff1371284542e48cbd467d3ef0ede9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.558332 12223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eec183c460ad4f1b85ad1407c53f2eab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.594900 12132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e393e90875847809e1ef94179c5b4ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.600317 12230 tablet_service.cc:1467] Processing CreateTablet for tablet 4b5f5f3a393f488ca713b7269b5dfa78 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260808 20:13:22.528637 12231 tablet_service.cc:1467] Processing CreateTablet for tablet a9ef909a266f4100bd54d95062fc7e25 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260808 20:13:22.558034 12237 tablet_service.cc:1467] Processing CreateTablet for tablet ddb83ac6240244be967995e7771a8b64 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260808 20:13:22.602330 12124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a383bcab83014009b5698c63cdac298b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.602380 12186 heartbeater.cc:503] Master 127.10.24.62:36417 was elected leader, sending a full tablet report...
I20260808 20:13:22.610316 12313 tablet_bootstrap.cc:492] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:22.614758 12228 tablet_service.cc:1467] Processing CreateTablet for tablet e5dc2ab0712f4054883178f4b4c50928 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260808 20:13:22.616252 12229 tablet_service.cc:1467] Processing CreateTablet for tablet d092815e2e3e40ba8cbcc396cbeeb3b7 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260808 20:13:22.618854 12132 tablet_service.cc:1467] Processing CreateTablet for tablet b0f8a2331e88407c9e3da24439d202b2 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260808 20:13:22.619994 12134 tablet_service.cc:1467] Processing CreateTablet for tablet c4dc15f09b4a4470ac96054b70602187 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260808 20:13:22.622231 12309 ts_tablet_manager.cc:1428] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.166s user 0.000s sys 0.015s
I20260808 20:13:22.638226 12030 tablet_service.cc:1467] Processing CreateTablet for tablet 4e5943c3d789423691d3fe7099372c59 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260808 20:13:22.639272 12132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f8a2331e88407c9e3da24439d202b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.639425 12030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e5943c3d789423691d3fe7099372c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.641243 12128 tablet_service.cc:1467] Processing CreateTablet for tablet 41bae2134ac343ed9e9f9c1b86703d97 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260808 20:13:22.646454 12033 tablet_service.cc:1467] Processing CreateTablet for tablet 2a41c716cfc94652a2fc6957a1bd2a78 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260808 20:13:22.641196 12313 ts_tablet_manager.cc:1397] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.193s user 0.013s sys 0.003s
I20260808 20:13:22.647673 12033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a41c716cfc94652a2fc6957a1bd2a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.646934 12034 tablet_service.cc:1467] Processing CreateTablet for tablet a383bcab83014009b5698c63cdac298b (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260808 20:13:22.650127 12034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a383bcab83014009b5698c63cdac298b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.555975 12125 tablet_service.cc:1467] Processing CreateTablet for tablet 6b217f6c1ea849f89667b8eb2d857206 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260808 20:13:22.653884 12313 raft_consensus.cc:348] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:22.654647 12313 raft_consensus.cc:374] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:22.654932 12313 raft_consensus.cc:729] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:22.655699 12313 consensus_queue.cc:260] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:22.657186 12309 tablet_bootstrap.cc:492] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:22.662483 12311 tablet_bootstrap.cc:492] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:22.662910 12311 ts_tablet_manager.cc:1397] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.282s user 0.025s sys 0.022s
I20260808 20:13:22.664394 12125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b217f6c1ea849f89667b8eb2d857206. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.675352 12313 ts_tablet_manager.cc:1428] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.028s user 0.005s sys 0.005s
I20260808 20:13:22.675798 12230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b5f5f3a393f488ca713b7269b5dfa78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.676796 12224 tablet_service.cc:1467] Processing CreateTablet for tablet 9d6f3bbe5e92475487be203c7c89b73d (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260808 20:13:22.680996 12134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4dc15f09b4a4470ac96054b70602187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.678826 12225 tablet_service.cc:1467] Processing CreateTablet for tablet 0be6c610fe68472b8bcb6232b862dbc1 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260808 20:13:22.682260 12231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ef909a266f4100bd54d95062fc7e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.692235 12288 heartbeater.cc:503] Master 127.10.24.62:36417 was elected leader, sending a full tablet report...
I20260808 20:13:22.693535 12030 tablet_service.cc:1467] Processing CreateTablet for tablet 6b217f6c1ea849f89667b8eb2d857206 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260808 20:13:22.694705 12030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b217f6c1ea849f89667b8eb2d857206. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.699033 12224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d6f3bbe5e92475487be203c7c89b73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.701395 12229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d092815e2e3e40ba8cbcc396cbeeb3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.700490 12136 tablet_service.cc:1467] Processing CreateTablet for tablet 2cd3d6998af44e69a7182125e02f16e2 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260808 20:13:22.708678 12128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41bae2134ac343ed9e9f9c1b86703d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.713266 12237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddb83ac6240244be967995e7771a8b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.714821 12084 heartbeater.cc:503] Master 127.10.24.62:36417 was elected leader, sending a full tablet report...
I20260808 20:13:22.715829 12230 tablet_service.cc:1467] Processing CreateTablet for tablet 7aff1371284542e48cbd467d3ef0ede9 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260808 20:13:22.717222 12230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aff1371284542e48cbd467d3ef0ede9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.718437 12225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be6c610fe68472b8bcb6232b862dbc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.687562 12031 tablet_service.cc:1467] Processing CreateTablet for tablet 85cecc37ef974ec58b1d5fe18e9801a2 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260808 20:13:22.741822 12136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cd3d6998af44e69a7182125e02f16e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.746747 12126 tablet_service.cc:1467] Processing CreateTablet for tablet 003bc27b07fa40e48c1e57a13b578981 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260808 20:13:22.741465 12228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5dc2ab0712f4054883178f4b4c50928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.755090 12126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 003bc27b07fa40e48c1e57a13b578981. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.726177 12313 tablet_bootstrap.cc:492] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:22.761163 12320 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:22.762967 12320 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:22.765974 12123 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7d7a85e3964f3f9d5f2dc39e23340b (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260808 20:13:22.767200 12123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7d7a85e3964f3f9d5f2dc39e23340b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.686993 12311 raft_consensus.cc:348] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:22.769323 12311 raft_consensus.cc:374] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:22.769625 12311 raft_consensus.cc:729] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:22.770332 12313 tablet_bootstrap.cc:654] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:22.734809 12230 tablet_service.cc:1467] Processing CreateTablet for tablet 2a41c716cfc94652a2fc6957a1bd2a78 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260808 20:13:22.746775 12127 tablet_service.cc:1467] Processing CreateTablet for tablet e912d871fe214ff9bbeacaa32fd5136b (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260808 20:13:22.757053 12234 tablet_service.cc:1467] Processing CreateTablet for tablet 5e931b9df2c74729a3771d5f0734f559 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260808 20:13:22.759050 12223 tablet_service.cc:1467] Processing CreateTablet for tablet 4e5943c3d789423691d3fe7099372c59 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260808 20:13:22.745959 12023 tablet_service.cc:1467] Processing CreateTablet for tablet 4e393e90875847809e1ef94179c5b4ef (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260808 20:13:22.785181 12309 tablet_bootstrap.cc:654] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:22.761080 12229 tablet_service.cc:1467] Processing CreateTablet for tablet a83f6daa15a54982865040c0110bdda8 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260808 20:13:22.796838 12132 tablet_service.cc:1467] Processing CreateTablet for tablet c0f516c855714940a41eac002203e145 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260808 20:13:22.803243 12024 tablet_service.cc:1467] Processing CreateTablet for tablet c4dc15f09b4a4470ac96054b70602187 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260808 20:13:22.797866 12127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e912d871fe214ff9bbeacaa32fd5136b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.805343 12034 tablet_service.cc:1467] Processing CreateTablet for tablet 2cd3d6998af44e69a7182125e02f16e2 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260808 20:13:22.801337 12223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e5943c3d789423691d3fe7099372c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.808512 12225 tablet_service.cc:1467] Processing CreateTablet for tablet a383bcab83014009b5698c63cdac298b (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260808 20:13:22.810416 12225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a383bcab83014009b5698c63cdac298b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.801429 12028 tablet_service.cc:1467] Processing CreateTablet for tablet 003bc27b07fa40e48c1e57a13b578981 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260808 20:13:22.816872 12229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83f6daa15a54982865040c0110bdda8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.817452 12132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0f516c855714940a41eac002203e145. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.813104 12320 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:22.826421 12031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85cecc37ef974ec58b1d5fe18e9801a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.837806 12025 tablet_service.cc:1467] Processing CreateTablet for tablet 41bae2134ac343ed9e9f9c1b86703d97 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260808 20:13:22.837819 12035 tablet_service.cc:1467] Processing CreateTablet for tablet e912d871fe214ff9bbeacaa32fd5136b (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260808 20:13:22.839673 12033 tablet_service.cc:1467] Processing CreateTablet for tablet b0f8a2331e88407c9e3da24439d202b2 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260808 20:13:22.824348 12235 tablet_service.cc:1467] Processing CreateTablet for tablet 85cecc37ef974ec58b1d5fe18e9801a2 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260808 20:13:22.845036 12235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85cecc37ef974ec58b1d5fe18e9801a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.790768 12311 consensus_queue.cc:260] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:22.853873 12034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cd3d6998af44e69a7182125e02f16e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.857172 12033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f8a2331e88407c9e3da24439d202b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.858001 12035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e912d871fe214ff9bbeacaa32fd5136b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.858191 12234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e931b9df2c74729a3771d5f0734f559. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.881150 12025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41bae2134ac343ed9e9f9c1b86703d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.881621 12230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a41c716cfc94652a2fc6957a1bd2a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.885218 12023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e393e90875847809e1ef94179c5b4ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.886468 12124 tablet_service.cc:1467] Processing CreateTablet for tablet 2607c6f53fc243d2abb6b2f74b6e2aa6 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260808 20:13:22.887656 12124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2607c6f53fc243d2abb6b2f74b6e2aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.890324 12313 tablet_bootstrap.cc:492] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:22.890780 12313 ts_tablet_manager.cc:1397] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.165s user 0.011s sys 0.011s
I20260808 20:13:22.897291 12028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 003bc27b07fa40e48c1e57a13b578981. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.898906 12125 tablet_service.cc:1467] Processing CreateTablet for tablet 6088cb69cb3c4d66afe9dc98da9179ea (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260808 20:13:22.900123 12125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6088cb69cb3c4d66afe9dc98da9179ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.904179 12024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4dc15f09b4a4470ac96054b70602187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.907799 12131 tablet_service.cc:1467] Processing CreateTablet for tablet 3051f9a615344b5c80595ef0ace350c8 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
W20260808 20:13:22.914906 11875 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 55514.8us
I20260808 20:13:22.922901 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:22.923121 12313 raft_consensus.cc:348] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } }
I20260808 20:13:22.923974 12313 raft_consensus.cc:374] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:22.924314 12313 raft_consensus.cc:729] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
W20260808 20:13:22.924557 12205 leader_election.cc:341] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:22.925046 12313 consensus_queue.cc:260] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } }
I20260808 20:13:22.928861 12022 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7d7a85e3964f3f9d5f2dc39e23340b (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260808 20:13:22.838677 12224 tablet_service.cc:1467] Processing CreateTablet for tablet 6b217f6c1ea849f89667b8eb2d857206 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260808 20:13:22.931773 12224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b217f6c1ea849f89667b8eb2d857206. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.935400 12131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3051f9a615344b5c80595ef0ace350c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.941557 12022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7d7a85e3964f3f9d5f2dc39e23340b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.950340 12023 tablet_service.cc:1467] Processing CreateTablet for tablet c0f516c855714940a41eac002203e145 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260808 20:13:22.951542 12023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0f516c855714940a41eac002203e145. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:22.960880 12320 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:22.961355 12320 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } }
I20260808 20:13:22.963233 12320 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:13:22.964305 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:22.965569 12319 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:22.966004 12319 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } }
W20260808 20:13:22.966563 12205 leader_election.cc:341] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:22.971231 12319 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:13:22.971036 12127 tablet_service.cc:1467] Processing CreateTablet for tablet 940aa50483d844deb4031f312b4a7355 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260808 20:13:22.965986 12313 ts_tablet_manager.cc:1428] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.075s user 0.004s sys 0.004s
I20260808 20:13:22.957934 12136 tablet_service.cc:1467] Processing CreateTablet for tablet 5fdbdda415914f1a9652ef68d1bba7fb (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260808 20:13:22.990746 12127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 940aa50483d844deb4031f312b4a7355. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.006079 12330 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:23.006460 12313 tablet_bootstrap.cc:492] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:23.006558 12330 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.012048 12313 tablet_bootstrap.cc:654] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:23.012995 12237 tablet_service.cc:1467] Processing CreateTablet for tablet 4e393e90875847809e1ef94179c5b4ef (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260808 20:13:23.014213 12237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e393e90875847809e1ef94179c5b4ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.015774 12309 tablet_bootstrap.cc:492] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:23.018720 12223 tablet_service.cc:1467] Processing CreateTablet for tablet b0f8a2331e88407c9e3da24439d202b2 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260808 20:13:23.019874 12223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f8a2331e88407c9e3da24439d202b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.030956 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:13:23.032459 12206 leader_election.cc:341] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:23.032859 12206 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:23.028254 12309 ts_tablet_manager.cc:1397] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.371s user 0.009s sys 0.018s
I20260808 20:13:23.034068 12320 raft_consensus.cc:2726] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:23.039961 12025 tablet_service.cc:1467] Processing CreateTablet for tablet 3051f9a615344b5c80595ef0ace350c8 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260808 20:13:23.041180 12025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3051f9a615344b5c80595ef0ace350c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.026758 12228 tablet_service.cc:1467] Processing CreateTablet for tablet c4dc15f09b4a4470ac96054b70602187 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260808 20:13:23.051200 12128 tablet_service.cc:1467] Processing CreateTablet for tablet 926cdce6649c4ca59961cfac78392647 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260808 20:13:23.035226 12330 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:23.052608 12134 tablet_service.cc:1467] Processing CreateTablet for tablet 67b75df4537844c2b76699830dedad62 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260808 20:13:23.057621 12136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fdbdda415914f1a9652ef68d1bba7fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.057590 12235 tablet_service.cc:1467] Processing CreateTablet for tablet 003bc27b07fa40e48c1e57a13b578981 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260808 20:13:23.058851 12235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 003bc27b07fa40e48c1e57a13b578981. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.061391 12134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b75df4537844c2b76699830dedad62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.061853 12033 tablet_service.cc:1467] Processing CreateTablet for tablet 2607c6f53fc243d2abb6b2f74b6e2aa6 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260808 20:13:23.051261 12311 ts_tablet_manager.cc:1428] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.387s user 0.014s sys 0.014s
I20260808 20:13:23.069052 12231 tablet_service.cc:1467] Processing CreateTablet for tablet e912d871fe214ff9bbeacaa32fd5136b (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260808 20:13:23.070236 12231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e912d871fe214ff9bbeacaa32fd5136b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.073349 12033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2607c6f53fc243d2abb6b2f74b6e2aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.077621 12311 tablet_bootstrap.cc:492] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:23.089485 12131 tablet_service.cc:1467] Processing CreateTablet for tablet e020c14c3aeb419493cd543071ed7801 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260808 20:13:23.036079 12126 tablet_service.cc:1467] Processing CreateTablet for tablet faebf2544dae4e21b7519aeeb8b250dc (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260808 20:13:23.090988 12128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 926cdce6649c4ca59961cfac78392647. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.094218 12234 tablet_service.cc:1467] Processing CreateTablet for tablet 41bae2134ac343ed9e9f9c1b86703d97 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260808 20:13:23.095504 12234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41bae2134ac343ed9e9f9c1b86703d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.098186 12237 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7d7a85e3964f3f9d5f2dc39e23340b (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260808 20:13:23.098333 12225 tablet_service.cc:1467] Processing CreateTablet for tablet 3051f9a615344b5c80595ef0ace350c8 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260808 20:13:23.099387 12237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7d7a85e3964f3f9d5f2dc39e23340b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.098827 12224 tablet_service.cc:1467] Processing CreateTablet for tablet c0f516c855714940a41eac002203e145 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260808 20:13:23.100119 12230 tablet_service.cc:1467] Processing CreateTablet for tablet 2607c6f53fc243d2abb6b2f74b6e2aa6 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260808 20:13:23.104442 12126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faebf2544dae4e21b7519aeeb8b250dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.106076 12224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0f516c855714940a41eac002203e145. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.106982 12230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2607c6f53fc243d2abb6b2f74b6e2aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.115849 12225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3051f9a615344b5c80595ef0ace350c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.122457 12123 tablet_service.cc:1467] Processing CreateTablet for tablet e256c29dec964f49856cf204f950793b (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260808 20:13:23.124423 12132 tablet_service.cc:1467] Processing CreateTablet for tablet 21889249378248049ef043f0588cebc9 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260808 20:13:23.128756 12031 tablet_service.cc:1467] Processing CreateTablet for tablet 940aa50483d844deb4031f312b4a7355 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260808 20:13:23.129091 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:23.129799 12150 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 0.
I20260808 20:13:23.134253 12228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4dc15f09b4a4470ac96054b70602187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.130025 12031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 940aa50483d844deb4031f312b4a7355. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.136819 12131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e020c14c3aeb419493cd543071ed7801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.141561 12028 tablet_service.cc:1467] Processing CreateTablet for tablet 926cdce6649c4ca59961cfac78392647 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260808 20:13:23.142920 12024 tablet_service.cc:1467] Processing CreateTablet for tablet faebf2544dae4e21b7519aeeb8b250dc (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260808 20:13:23.144933 12124 tablet_service.cc:1467] Processing CreateTablet for tablet fbc85c9f50234a49833e25d612f66f96 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260808 20:13:23.146821 12134 tablet_service.cc:1467] Processing CreateTablet for tablet a16032fd7a244ded8d9660c183645279 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260808 20:13:23.147112 12309 raft_consensus.cc:348] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.159780 12229 tablet_service.cc:1467] Processing CreateTablet for tablet 5fdbdda415914f1a9652ef68d1bba7fb (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260808 20:13:23.149045 12206 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:23.162026 12320 raft_consensus.cc:2781] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:23.162379 12320 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:23.162683 12320 raft_consensus.cc:3037] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:23.165400 12028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 926cdce6649c4ca59961cfac78392647. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.168376 12034 tablet_service.cc:1467] Processing CreateTablet for tablet 6088cb69cb3c4d66afe9dc98da9179ea (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260808 20:13:23.172108 12023 tablet_service.cc:1467] Processing CreateTablet for tablet 67b75df4537844c2b76699830dedad62 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260808 20:13:23.176625 12311 tablet_bootstrap.cc:654] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:23.166667 12309 raft_consensus.cc:374] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:23.181200 12309 raft_consensus.cc:729] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:23.182014 12309 consensus_queue.cc:260] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.182793 12125 tablet_service.cc:1467] Processing CreateTablet for tablet 1e0d98ab3b4945d8b845711f4ddd7153 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260808 20:13:23.183564 12132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21889249378248049ef043f0588cebc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.185350 12123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e256c29dec964f49856cf204f950793b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.188066 12134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a16032fd7a244ded8d9660c183645279. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.191642 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:13:23.193147 12102 leader_election.cc:341] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:23.195144 12320 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } }
I20260808 20:13:23.167276 12229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fdbdda415914f1a9652ef68d1bba7fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.197974 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:23.198521 12150 raft_consensus.cc:3037] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:23.199899 12125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e0d98ab3b4945d8b845711f4ddd7153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.201167 12124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbc85c9f50234a49833e25d612f66f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.204509 12034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6088cb69cb3c4d66afe9dc98da9179ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.207098 12309 ts_tablet_manager.cc:1428] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.164s user 0.003s sys 0.003s
I20260808 20:13:23.208026 12309 tablet_bootstrap.cc:492] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:23.207969 12022 tablet_service.cc:1467] Processing CreateTablet for tablet e020c14c3aeb419493cd543071ed7801 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260808 20:13:23.209221 12022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e020c14c3aeb419493cd543071ed7801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.211092 12035 tablet_service.cc:1467] Processing CreateTablet for tablet e256c29dec964f49856cf204f950793b (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260808 20:13:23.212215 12035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e256c29dec964f49856cf204f950793b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.213171 12023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b75df4537844c2b76699830dedad62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.214129 12223 tablet_service.cc:1467] Processing CreateTablet for tablet 6088cb69cb3c4d66afe9dc98da9179ea (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260808 20:13:23.215315 12223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6088cb69cb3c4d66afe9dc98da9179ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.217383 12024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faebf2544dae4e21b7519aeeb8b250dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.222499 12313 tablet_bootstrap.cc:492] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:23.147883 12030 tablet_service.cc:1467] Processing CreateTablet for tablet 5fdbdda415914f1a9652ef68d1bba7fb (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260808 20:13:23.223703 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:23.224077 12150 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
W20260808 20:13:23.225282 11994 leader_election.cc:341] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:23.231992 12025 tablet_service.cc:1467] Processing CreateTablet for tablet a16032fd7a244ded8d9660c183645279 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260808 20:13:23.223919 12030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fdbdda415914f1a9652ef68d1bba7fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.237809 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:13:23.239337 12000 leader_election.cc:341] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:23.239794 12000 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:23.240334 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:23.240649 12330 raft_consensus.cc:2726] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:23.240984 12206 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
W20260808 20:13:23.241770 12205 leader_election.cc:341] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:23.241820 12321 raft_consensus.cc:2781] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:23.242542 12320 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:13:23.245193 12025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a16032fd7a244ded8d9660c183645279. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.238181 12136 tablet_service.cc:1467] Processing CreateTablet for tablet aa97e9c654da4bcda377ef63ec1c694b (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260808 20:13:23.254124 12136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa97e9c654da4bcda377ef63ec1c694b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.253206 12313 ts_tablet_manager.cc:1397] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.217s user 0.012s sys 0.024s
I20260808 20:13:23.271802 12313 raft_consensus.cc:348] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.272603 12313 raft_consensus.cc:374] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:23.279893 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:23.272886 12313 raft_consensus.cc:729] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:23.308796 12223 tablet_service.cc:1467] Processing CreateTablet for tablet 67b75df4537844c2b76699830dedad62 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260808 20:13:23.310410 12223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b75df4537844c2b76699830dedad62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.310076 12313 consensus_queue.cc:260] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.311403 12224 tablet_service.cc:1467] Processing CreateTablet for tablet e020c14c3aeb419493cd543071ed7801 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260808 20:13:23.312561 12224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e020c14c3aeb419493cd543071ed7801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.289667 12235 tablet_service.cc:1467] Processing CreateTablet for tablet 940aa50483d844deb4031f312b4a7355 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260808 20:13:23.301663 12028 tablet_service.cc:1467] Processing CreateTablet for tablet 21889249378248049ef043f0588cebc9 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260808 20:13:23.315265 12028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21889249378248049ef043f0588cebc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.282156 12309 tablet_bootstrap.cc:654] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:23.330204 12126 tablet_service.cc:1467] Processing CreateTablet for tablet ce978b5a06de44eabba785178c7a0100 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260808 20:13:23.319590 12320 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:23.312554 12313 ts_tablet_manager.cc:1428] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.051s user 0.005s sys 0.001s
I20260808 20:13:23.331364 12126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce978b5a06de44eabba785178c7a0100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.334190 12235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 940aa50483d844deb4031f312b4a7355. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.335374 12123 tablet_service.cc:1467] Processing CreateTablet for tablet efc388dadfe84be190c9ddd0b3725244 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260808 20:13:23.336529 12123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efc388dadfe84be190c9ddd0b3725244. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.282732 12230 tablet_service.cc:1467] Processing CreateTablet for tablet 926cdce6649c4ca59961cfac78392647 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260808 20:13:23.330960 12320 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.342813 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:23.341806 12320 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:13:23.343961 12206 leader_election.cc:341] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:23.344310 12127 tablet_service.cc:1467] Processing CreateTablet for tablet 102eea36004446e5a71aaccebb9ad859 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260808 20:13:23.345139 12231 tablet_service.cc:1467] Processing CreateTablet for tablet e256c29dec964f49856cf204f950793b (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260808 20:13:23.341784 12031 tablet_service.cc:1467] Processing CreateTablet for tablet fbc85c9f50234a49833e25d612f66f96 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260808 20:13:23.347131 12031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbc85c9f50234a49833e25d612f66f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.346328 12313 tablet_bootstrap.cc:492] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:23.337703 12033 tablet_service.cc:1467] Processing CreateTablet for tablet 1e0d98ab3b4945d8b845711f4ddd7153 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260808 20:13:23.337994 12311 tablet_bootstrap.cc:492] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:23.350759 12311 ts_tablet_manager.cc:1397] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.273s user 0.014s sys 0.014s
I20260808 20:13:23.279937 12237 tablet_service.cc:1467] Processing CreateTablet for tablet faebf2544dae4e21b7519aeeb8b250dc (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260808 20:13:23.345477 12127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 102eea36004446e5a71aaccebb9ad859. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.353770 12321 raft_consensus.cc:686] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:23.354681 12225 tablet_service.cc:1467] Processing CreateTablet for tablet a16032fd7a244ded8d9660c183645279 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260808 20:13:23.354648 12321 consensus_queue.cc:237] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } }
I20260808 20:13:23.353447 12313 tablet_bootstrap.cc:654] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:23.346259 12022 tablet_service.cc:1467] Processing CreateTablet for tablet aa97e9c654da4bcda377ef63ec1c694b (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260808 20:13:23.364722 12022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa97e9c654da4bcda377ef63ec1c694b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.361212 12231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e256c29dec964f49856cf204f950793b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.366746 12034 tablet_service.cc:1467] Processing CreateTablet for tablet 7cb737cc041b47a88cab93a3e6ed8158 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260808 20:13:23.368005 12229 tablet_service.cc:1467] Processing CreateTablet for tablet 21889249378248049ef043f0588cebc9 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260808 20:13:23.369258 12228 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44376 (request call id 28) took 1060 ms. Trace:
I20260808 20:13:23.369640 12228 rpcz_store.cc:276] 0808 20:13:22.309273 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:13:23.025787 (+716514us) service_pool.cc:227] Handling call
0808 20:13:23.122498 (+ 96711us) spinlock_profiling.cc:243] Waited 7.21 ms on lock 0x7b58001a1398. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d288d21 00007fa96d2661e3 00007fa96d217869 00007fa966819c44 00007fa966819bd2 00007fa966819b01
0808 20:13:23.122643 (+ 145us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0808 20:13:23.122743 (+ 100us) ts_tablet_manager.cc:676] Creating new metadata...
0808 20:13:23.179122 (+ 56379us) tablet_metadata.cc:641] Metadata flushed
0808 20:13:23.349711 (+170589us) tablet_replica.cc:205] Creating consensus instance
0808 20:13:23.369206 (+ 19495us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16580352}
I20260808 20:13:23.371891 12228 tablet_service.cc:1467] Processing CreateTablet for tablet fbc85c9f50234a49833e25d612f66f96 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260808 20:13:23.373350 12230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 926cdce6649c4ca59961cfac78392647. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.367936 12033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e0d98ab3b4945d8b845711f4ddd7153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.302512 12128 tablet_service.cc:1467] Processing CreateTablet for tablet 7cb737cc041b47a88cab93a3e6ed8158 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260808 20:13:23.377969 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:23.378582 12048 raft_consensus.cc:2454] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 0.
I20260808 20:13:23.378780 12229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21889249378248049ef043f0588cebc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.381053 12228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbc85c9f50234a49833e25d612f66f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.387094 12034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cb737cc041b47a88cab93a3e6ed8158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.389219 12237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faebf2544dae4e21b7519aeeb8b250dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.393817 12136 tablet_service.cc:1467] Processing CreateTablet for tablet b7e4c0cacf074dee848eed1ab4d27ebf (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260808 20:13:23.395948 12101 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:23.397121 12319 raft_consensus.cc:2726] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:23.398146 12128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cb737cc041b47a88cab93a3e6ed8158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.401593 12205 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:23.402056 12025 tablet_service.cc:1467] Processing CreateTablet for tablet ce978b5a06de44eabba785178c7a0100 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260808 20:13:23.402429 12321 raft_consensus.cc:2781] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:23.402765 12321 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:23.403070 12321 raft_consensus.cc:3037] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:23.403277 12025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce978b5a06de44eabba785178c7a0100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.403625 12125 tablet_service.cc:1467] Processing CreateTablet for tablet 0285ac4646ab40c3b248ff959bbcd41e (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260808 20:13:23.385057 12311 raft_consensus.cc:348] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.404603 12311 raft_consensus.cc:374] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:23.405354 12134 tablet_service.cc:1467] Processing CreateTablet for tablet 78ef4041ff114e34a9dff167914c95ef (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260808 20:13:23.410728 12321 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.412276 12311 raft_consensus.cc:729] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:23.413033 12311 consensus_queue.cc:260] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.413516 12321 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:23.413686 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:23.414288 12048 raft_consensus.cc:3037] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:23.415109 12311 ts_tablet_manager.cc:1428] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.064s user 0.006s sys 0.000s
I20260808 20:13:23.415968 12311 tablet_bootstrap.cc:492] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:23.417220 12225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a16032fd7a244ded8d9660c183645279. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.419710 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
W20260808 20:13:23.421422 12206 leader_election.cc:341] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:23.421449 12234 tablet_service.cc:1467] Processing CreateTablet for tablet aa97e9c654da4bcda377ef63ec1c694b (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260808 20:13:23.423547 12234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa97e9c654da4bcda377ef63ec1c694b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.419916 12228 tablet_service.cc:1467] Processing CreateTablet for tablet 1e0d98ab3b4945d8b845711f4ddd7153 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260808 20:13:23.426894 12228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e0d98ab3b4945d8b845711f4ddd7153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.442771 12309 tablet_bootstrap.cc:492] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:23.443272 12309 ts_tablet_manager.cc:1397] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.235s user 0.012s sys 0.005s
I20260808 20:13:23.373607 12131 tablet_service.cc:1467] Processing CreateTablet for tablet 237b38cfed904b8395749a75666444e9 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260808 20:13:23.460181 12309 raft_consensus.cc:348] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.460968 12309 raft_consensus.cc:374] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:23.461263 12309 raft_consensus.cc:729] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:23.426270 11892 catalog_manager.cc:5495] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 0 to 1, leader changed from <none> to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:23.404734 12125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0285ac4646ab40c3b248ff959bbcd41e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.471028 12048 raft_consensus.cc:2454] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:13:23.472190 12205 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:23.472968 12321 raft_consensus.cc:2781] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:23.473408 12321 raft_consensus.cc:686] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:23.474152 12321 consensus_queue.cc:237] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.476437 12023 tablet_service.cc:1467] Processing CreateTablet for tablet 102eea36004446e5a71aaccebb9ad859 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260808 20:13:23.477609 12023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 102eea36004446e5a71aaccebb9ad859. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.477314 12223 tablet_service.cc:1467] Processing CreateTablet for tablet 7cb737cc041b47a88cab93a3e6ed8158 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260808 20:13:23.478580 12223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cb737cc041b47a88cab93a3e6ed8158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.486052 12033 tablet_service.cc:1467] Processing CreateTablet for tablet 78ef4041ff114e34a9dff167914c95ef (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260808 20:13:23.491415 12034 tablet_service.cc:1467] Processing CreateTablet for tablet 237b38cfed904b8395749a75666444e9 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260808 20:13:23.493160 12024 tablet_service.cc:1467] Processing CreateTablet for tablet efc388dadfe84be190c9ddd0b3725244 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260808 20:13:23.493604 12124 tablet_service.cc:1467] Processing CreateTablet for tablet ac9505fd371049b7b3ca47aad176ad64 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260808 20:13:23.498718 12235 tablet_service.cc:1467] Processing CreateTablet for tablet ce978b5a06de44eabba785178c7a0100 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260808 20:13:23.505584 12235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce978b5a06de44eabba785178c7a0100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.499171 12034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 237b38cfed904b8395749a75666444e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.518890 12030 tablet_service.cc:1467] Processing CreateTablet for tablet b7e4c0cacf074dee848eed1ab4d27ebf (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260808 20:13:23.521229 12030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7e4c0cacf074dee848eed1ab4d27ebf. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 20:13:23.534186 11966 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 20:13:23.555109 12234 tablet_service.cc:1467] Processing CreateTablet for tablet 102eea36004446e5a71aaccebb9ad859 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260808 20:13:23.556643 12234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 102eea36004446e5a71aaccebb9ad859. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.524036 12134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78ef4041ff114e34a9dff167914c95ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.590193 12311 tablet_bootstrap.cc:654] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:23.591708 12234 tablet_service.cc:1467] Processing CreateTablet for tablet 78ef4041ff114e34a9dff167914c95ef (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260808 20:13:23.593170 12234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78ef4041ff114e34a9dff167914c95ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.596769 12124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9505fd371049b7b3ca47aad176ad64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.602576 12035 tablet_service.cc:1467] Processing CreateTablet for tablet ac9505fd371049b7b3ca47aad176ad64 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260808 20:13:23.603966 12035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9505fd371049b7b3ca47aad176ad64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.602571 12022 tablet_service.cc:1467] Processing CreateTablet for tablet 0285ac4646ab40c3b248ff959bbcd41e (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260808 20:13:23.605311 12022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0285ac4646ab40c3b248ff959bbcd41e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.616117 12237 tablet_service.cc:1467] Processing CreateTablet for tablet 237b38cfed904b8395749a75666444e9 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260808 20:13:23.617130 12237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 237b38cfed904b8395749a75666444e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.626019 12228 tablet_service.cc:1467] Processing CreateTablet for tablet efc388dadfe84be190c9ddd0b3725244 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260808 20:13:23.622876 12131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 237b38cfed904b8395749a75666444e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.629408 12136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7e4c0cacf074dee848eed1ab4d27ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.516861 12309 consensus_queue.cc:260] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.633677 12319 raft_consensus.cc:482] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:23.636582 12319 raft_consensus.cc:504] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.633805 12033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78ef4041ff114e34a9dff167914c95ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.648229 12228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efc388dadfe84be190c9ddd0b3725244. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.650077 12313 tablet_bootstrap.cc:492] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:23.650516 12313 ts_tablet_manager.cc:1397] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.304s user 0.067s sys 0.028s
I20260808 20:13:23.651228 12229 tablet_service.cc:1467] Processing CreateTablet for tablet 0285ac4646ab40c3b248ff959bbcd41e (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260808 20:13:23.652346 12229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0285ac4646ab40c3b248ff959bbcd41e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.626003 12223 tablet_service.cc:1467] Processing CreateTablet for tablet b7e4c0cacf074dee848eed1ab4d27ebf (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260808 20:13:23.653885 12223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7e4c0cacf074dee848eed1ab4d27ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.661944 12231 tablet_service.cc:1467] Processing CreateTablet for tablet ac9505fd371049b7b3ca47aad176ad64 (DEFAULT_TABLE table=test-workload [id=b2172fa8e7574b4990306664f4c37ada]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260808 20:13:23.663132 12231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9505fd371049b7b3ca47aad176ad64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:23.669294 12319 leader_election.cc:288] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:23.669766 12319 raft_consensus.cc:482] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:23.669965 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:23.670089 12319 raft_consensus.cc:504] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.671096 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:13:23.671293 12102 leader_election.cc:341] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:23.671890 12319 leader_election.cc:288] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:13:23.672374 12101 leader_election.cc:341] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:23.672765 12101 leader_election.cc:302] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:23.673753 12319 raft_consensus.cc:2726] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:23.673609 12313 raft_consensus.cc:348] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.674333 12313 raft_consensus.cc:374] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:23.674597 12313 raft_consensus.cc:729] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:23.679805 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:23.673547 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:23.681725 12024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efc388dadfe84be190c9ddd0b3725244. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 20:13:23.682322 12101 leader_election.cc:341] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
W20260808 20:13:23.684753 12102 leader_election.cc:341] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:23.684794 12230 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44376 (request call id 40) took 1053 ms. Trace:
I20260808 20:13:23.685326 12230 rpcz_store.cc:276] 0808 20:13:22.631513 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:13:23.277607 (+646094us) service_pool.cc:227] Handling call
0808 20:13:23.366486 (+ 88879us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0808 20:13:23.366596 (+ 110us) ts_tablet_manager.cc:676] Creating new metadata...
0808 20:13:23.488943 (+122347us) tablet_metadata.cc:641] Metadata flushed
0808 20:13:23.642291 (+153348us) tablet_replica.cc:205] Creating consensus instance
0808 20:13:23.684741 (+ 42450us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:13:23.723124 12224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44376 (request call id 43) took 1031 ms. Trace:
I20260808 20:13:23.723608 12224 rpcz_store.cc:276] 0808 20:13:22.691238 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:13:23.310636 (+619398us) service_pool.cc:227] Handling call
0808 20:13:23.312150 (+ 1514us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0808 20:13:23.312234 (+ 84us) ts_tablet_manager.cc:676] Creating new metadata...
0808 20:13:23.563488 (+251254us) tablet_metadata.cc:641] Metadata flushed
0808 20:13:23.722616 (+159128us) tablet_replica.cc:205] Creating consensus instance
0808 20:13:23.723064 (+ 448us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9317}
I20260808 20:13:23.724745 12313 consensus_queue.cc:260] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.728021 12313 ts_tablet_manager.cc:1428] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.077s user 0.004s sys 0.002s
I20260808 20:13:23.749827 12311 tablet_bootstrap.cc:492] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:23.685333 12102 leader_election.cc:302] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:23.751163 12313 tablet_bootstrap.cc:492] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:23.754730 12311 ts_tablet_manager.cc:1397] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.339s user 0.023s sys 0.002s
I20260808 20:13:23.755864 12309 ts_tablet_manager.cc:1428] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.312s user 0.046s sys 0.042s
I20260808 20:13:23.756846 12309 tablet_bootstrap.cc:492] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:23.761888 12311 raft_consensus.cc:348] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.764595 12311 raft_consensus.cc:374] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:23.764885 12311 raft_consensus.cc:729] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
W20260808 20:13:23.765026 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 80388.1us
I20260808 20:13:23.765708 12360 raft_consensus.cc:2726] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:23.765656 12311 consensus_queue.cc:260] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.767802 12311 ts_tablet_manager.cc:1428] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260808 20:13:23.773680 12309 tablet_bootstrap.cc:654] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:23.780922 12311 tablet_bootstrap.cc:492] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:23.782195 12313 tablet_bootstrap.cc:654] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:23.784703 12330 raft_consensus.cc:482] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:23.785168 12330 raft_consensus.cc:504] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.787117 12330 leader_election.cc:288] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:23.788795 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:13:23.790094 11994 leader_election.cc:341] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:23.791852 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:13:23.793025 12000 leader_election.cc:341] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:23.793438 12000 leader_election.cc:302] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:23.794390 12330 raft_consensus.cc:2726] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 20:13:23.796732 12205 consensus_peers.cc:487] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a -> Peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Couldn't send request to peer 4cd46f13bceb4c3c9ef71e755e17b995. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:23.802695 12309 tablet_bootstrap.cc:492] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:23.803021 12313 tablet_bootstrap.cc:492] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:23.803162 12309 ts_tablet_manager.cc:1397] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.003s
I20260808 20:13:23.803475 12313 ts_tablet_manager.cc:1397] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.054s user 0.012s sys 0.015s
I20260808 20:13:23.805763 12309 raft_consensus.cc:348] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.805990 12313 raft_consensus.cc:348] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.806479 12309 raft_consensus.cc:374] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:23.806725 12313 raft_consensus.cc:374] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:23.806806 12309 raft_consensus.cc:729] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:23.807101 12313 raft_consensus.cc:729] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:23.807286 12311 tablet_bootstrap.cc:654] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:23.807575 12309 consensus_queue.cc:260] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.807888 12313 consensus_queue.cc:260] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.810055 12313 ts_tablet_manager.cc:1428] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 20:13:23.809263 11892 catalog_manager.cc:5495] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 0 to 1, leader changed from <none> to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:13:23.810235 12309 ts_tablet_manager.cc:1428] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260808 20:13:23.810966 12313 tablet_bootstrap.cc:492] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:23.811293 12309 tablet_bootstrap.cc:492] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:23.817322 12313 tablet_bootstrap.cc:654] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:23.818913 12309 tablet_bootstrap.cc:654] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:23.821288 12359 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:23.821736 12359 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.823699 12359 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:23.825865 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:23.826064 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:23.826728 12254 raft_consensus.cc:2454] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
W20260808 20:13:23.827198 12102 leader_election.cc:341] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:23.828189 12101 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:23.829492 12359 raft_consensus.cc:2781] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:23.829843 12359 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:23.830159 12359 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:23.835422 12313 tablet_bootstrap.cc:492] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:23.835891 12313 ts_tablet_manager.cc:1397] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.025s user 0.018s sys 0.004s
I20260808 20:13:23.838562 12359 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.838644 12313 raft_consensus.cc:348] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.839468 12313 raft_consensus.cc:374] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:23.839774 12313 raft_consensus.cc:729] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:23.840449 12313 consensus_queue.cc:260] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.841388 12309 tablet_bootstrap.cc:492] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:23.841820 12309 ts_tablet_manager.cc:1397] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.007s
I20260808 20:13:23.842231 12311 tablet_bootstrap.cc:492] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:23.842631 12313 ts_tablet_manager.cc:1428] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.006s user 0.001s sys 0.006s
I20260808 20:13:23.842653 12311 ts_tablet_manager.cc:1397] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.063s user 0.004s sys 0.009s
I20260808 20:13:23.841645 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:23.843889 12313 tablet_bootstrap.cc:492] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:23.844403 12309 raft_consensus.cc:348] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.845110 12309 raft_consensus.cc:374] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:23.845436 12309 raft_consensus.cc:729] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:23.845299 12311 raft_consensus.cc:348] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } }
I20260808 20:13:23.846124 12311 raft_consensus.cc:374] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:23.846478 12311 raft_consensus.cc:729] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:23.846355 12309 consensus_queue.cc:260] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:13:23.844347 12102 leader_election.cc:341] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:23.848842 12309 ts_tablet_manager.cc:1428] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 20:13:23.850123 12309 tablet_bootstrap.cc:492] T 72e59eff611e4cc693407979ebe0857c P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:23.850489 12311 consensus_queue.cc:260] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } }
I20260808 20:13:23.851184 12313 tablet_bootstrap.cc:654] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:23.852885 12311 ts_tablet_manager.cc:1428] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260808 20:13:23.853945 12311 tablet_bootstrap.cc:492] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:23.855981 12309 tablet_bootstrap.cc:654] T 72e59eff611e4cc693407979ebe0857c P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:23.859637 12311 tablet_bootstrap.cc:654] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:23.859472 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:23.860080 12254 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:23.860886 12321 consensus_queue.cc:1035] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:23.866968 12254 raft_consensus.cc:2454] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:23.867210 12309 tablet_bootstrap.cc:492] T 72e59eff611e4cc693407979ebe0857c P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:23.867615 12309 ts_tablet_manager.cc:1397] T 72e59eff611e4cc693407979ebe0857c P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.008s
I20260808 20:13:23.867997 12101 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:23.870261 12360 raft_consensus.cc:2781] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:23.870353 12309 raft_consensus.cc:348] T 72e59eff611e4cc693407979ebe0857c P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.870767 12360 raft_consensus.cc:686] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:23.871183 12359 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:23.871587 12360 consensus_queue.cc:237] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.875396 12309 raft_consensus.cc:374] T 72e59eff611e4cc693407979ebe0857c P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:23.875703 12309 raft_consensus.cc:729] T 72e59eff611e4cc693407979ebe0857c P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:23.876507 12309 consensus_queue.cc:260] T 72e59eff611e4cc693407979ebe0857c P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.878799 12309 ts_tablet_manager.cc:1428] T 72e59eff611e4cc693407979ebe0857c P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260808 20:13:23.879684 12309 tablet_bootstrap.cc:492] T 4e4bf0005d9e43c1a4adce7588e7653f P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:23.882071 12311 tablet_bootstrap.cc:492] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:23.882555 12311 ts_tablet_manager.cc:1397] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.000s
I20260808 20:13:23.885354 12309 tablet_bootstrap.cc:654] T 4e4bf0005d9e43c1a4adce7588e7653f P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:23.885118 12311 raft_consensus.cc:348] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.885810 12311 raft_consensus.cc:374] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:23.886088 12311 raft_consensus.cc:729] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:23.887360 12311 consensus_queue.cc:260] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.890383 12313 tablet_bootstrap.cc:492] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:23.890864 12313 ts_tablet_manager.cc:1397] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.047s user 0.007s sys 0.015s
I20260808 20:13:23.891301 12309 tablet_bootstrap.cc:492] T 4e4bf0005d9e43c1a4adce7588e7653f P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:23.891706 12309 ts_tablet_manager.cc:1397] T 4e4bf0005d9e43c1a4adce7588e7653f P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260808 20:13:23.892725 12311 ts_tablet_manager.cc:1428] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.010s user 0.004s sys 0.001s
I20260808 20:13:23.893646 12311 tablet_bootstrap.cc:492] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:23.894171 12309 raft_consensus.cc:348] T 4e4bf0005d9e43c1a4adce7588e7653f P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.894838 12309 raft_consensus.cc:374] T 4e4bf0005d9e43c1a4adce7588e7653f P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:23.895118 12309 raft_consensus.cc:729] T 4e4bf0005d9e43c1a4adce7588e7653f P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:23.895888 12309 consensus_queue.cc:260] T 4e4bf0005d9e43c1a4adce7588e7653f P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.898345 12309 ts_tablet_manager.cc:1428] T 4e4bf0005d9e43c1a4adce7588e7653f P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260808 20:13:23.899235 12309 tablet_bootstrap.cc:492] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:23.899652 12311 tablet_bootstrap.cc:654] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:23.901643 12330 raft_consensus.cc:482] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:23.902070 12330 raft_consensus.cc:504] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.903973 12330 leader_election.cc:288] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:23.904711 12321 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:23.905169 12321 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.906159 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:23.906293 12313 raft_consensus.cc:348] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.906769 12150 raft_consensus.cc:2454] T 4e4bf0005d9e43c1a4adce7588e7653f P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:23.906988 12313 raft_consensus.cc:374] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:23.907332 12313 raft_consensus.cc:729] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:23.907701 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:23.907362 12321 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:23.908103 12309 tablet_bootstrap.cc:654] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:23.908031 12313 consensus_queue.cc:260] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.908988 12000 leader_election.cc:302] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:13:23.909236 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:13:23.910415 11994 leader_election.cc:341] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:23.910818 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:23.911270 12330 raft_consensus.cc:2781] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:23.911643 12330 raft_consensus.cc:482] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:23.911947 12330 raft_consensus.cc:3037] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:23.912559 12313 ts_tablet_manager.cc:1428] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.021s user 0.008s sys 0.000s
W20260808 20:13:23.913190 12206 leader_election.cc:341] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:23.913465 12313 tablet_bootstrap.cc:492] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
W20260808 20:13:23.914728 12205 leader_election.cc:341] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:23.915135 12205 leader_election.cc:302] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:23.919453 12313 tablet_bootstrap.cc:654] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:23.921386 12330 raft_consensus.cc:504] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.922709 12321 raft_consensus.cc:2726] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:23.924198 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:23.924765 12150 raft_consensus.cc:3037] T 4e4bf0005d9e43c1a4adce7588e7653f P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:23.925386 11892 catalog_manager.cc:5495] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:23.931058 12313 tablet_bootstrap.cc:492] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:23.931478 12313 ts_tablet_manager.cc:1397] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.003s
I20260808 20:13:23.934110 12313 raft_consensus.cc:348] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.934912 12313 raft_consensus.cc:374] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:23.935279 12313 raft_consensus.cc:729] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:23.935894 12313 consensus_queue.cc:260] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.936827 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:23.937444 12254 raft_consensus.cc:3037] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:23.937768 12313 ts_tablet_manager.cc:1428] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 20:13:23.938616 12313 tablet_bootstrap.cc:492] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:23.939558 12321 raft_consensus.cc:482] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:23.939978 12321 raft_consensus.cc:504] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.941341 12349 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:23.941466 12321 leader_election.cc:288] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:23.934813 12150 raft_consensus.cc:2454] T 4e4bf0005d9e43c1a4adce7588e7653f P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:23.941828 12349 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.942209 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:23.942819 12048 raft_consensus.cc:2454] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 0.
I20260808 20:13:23.943886 12349 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:23.944880 12205 leader_election.cc:302] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:23.946497 12313 tablet_bootstrap.cc:654] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:23.946619 12000 leader_election.cc:302] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:13:23.948498 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:23.949360 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:13:23.949851 12205 leader_election.cc:341] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:23.950425 12349 raft_consensus.cc:2781] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:23.950778 12349 raft_consensus.cc:482] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:23.951097 12349 raft_consensus.cc:3037] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:23.951512 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:23.952184 12150 raft_consensus.cc:2454] T 72e59eff611e4cc693407979ebe0857c P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 0.
W20260808 20:13:23.957171 12206 leader_election.cc:341] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:23.957633 12206 leader_election.cc:302] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:23.958364 12321 raft_consensus.cc:2726] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:23.956756 12254 raft_consensus.cc:2454] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:23.965279 12384 raft_consensus.cc:2781] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:23.966782 12330 leader_election.cc:288] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:23.966889 12349 raft_consensus.cc:504] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.968456 12384 raft_consensus.cc:686] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:23.970403 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:23.971024 12048 raft_consensus.cc:3037] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:23.973842 12311 tablet_bootstrap.cc:492] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:23.974375 12311 ts_tablet_manager.cc:1397] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.081s user 0.008s sys 0.004s
I20260808 20:13:23.977314 12311 raft_consensus.cc:348] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.978313 12311 raft_consensus.cc:374] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:23.978682 12311 raft_consensus.cc:729] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:23.979638 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:23.980237 12150 raft_consensus.cc:3037] T 72e59eff611e4cc693407979ebe0857c P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:23.979445 12311 consensus_queue.cc:260] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.982751 12309 tablet_bootstrap.cc:492] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:23.983348 12309 ts_tablet_manager.cc:1397] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.084s user 0.033s sys 0.041s
I20260808 20:13:23.984694 12311 ts_tablet_manager.cc:1428] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.010s user 0.001s sys 0.004s
I20260808 20:13:23.985697 12311 tablet_bootstrap.cc:492] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:23.985836 12309 raft_consensus.cc:348] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.986500 12309 raft_consensus.cc:374] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:23.986753 12309 raft_consensus.cc:729] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:23.986996 12150 raft_consensus.cc:2454] T 72e59eff611e4cc693407979ebe0857c P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:13:23.987787 12313 tablet_bootstrap.cc:492] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:23.970227 12384 consensus_queue.cc:237] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.987358 12309 consensus_queue.cc:260] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.988206 12313 ts_tablet_manager.cc:1397] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.020s
I20260808 20:13:23.988085 12206 leader_election.cc:302] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:23.989157 12321 raft_consensus.cc:2781] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:23.990510 12313 raft_consensus.cc:348] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.991163 12313 raft_consensus.cc:374] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:23.991417 12313 raft_consensus.cc:729] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:23.992225 12309 ts_tablet_manager.cc:1428] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260808 20:13:23.992015 12313 consensus_queue.cc:260] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:23.993069 12309 tablet_bootstrap.cc:492] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:24.002846 12309 tablet_bootstrap.cc:654] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.005972 12349 leader_election.cc:288] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:23.977795 12048 raft_consensus.cc:2454] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:13:24.007994 12311 tablet_bootstrap.cc:654] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.012526 12321 raft_consensus.cc:686] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:24.013480 12321 consensus_queue.cc:237] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.017284 12309 tablet_bootstrap.cc:492] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:24.017737 12309 ts_tablet_manager.cc:1397] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.005s
I20260808 20:13:24.018333 11892 catalog_manager.cc:5495] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:24.020090 12309 raft_consensus.cc:348] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.020771 12309 raft_consensus.cc:374] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.021057 12309 raft_consensus.cc:729] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.021903 12309 consensus_queue.cc:260] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.023106 12313 ts_tablet_manager.cc:1428] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.035s user 0.019s sys 0.011s
I20260808 20:13:24.023931 12313 tablet_bootstrap.cc:492] T f4918de0ffe94198a66ac1975ddbc693 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:24.028398 12384 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:24.029124 12384 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } }
I20260808 20:13:24.029847 12309 ts_tablet_manager.cc:1428] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260808 20:13:24.030778 12309 tablet_bootstrap.cc:492] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:24.031164 12384 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.032766 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:24.034590 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:24.036674 12000 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:24.040273 12313 tablet_bootstrap.cc:654] T f4918de0ffe94198a66ac1975ddbc693 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.038682 12384 raft_consensus.cc:2726] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:24.041254 12309 tablet_bootstrap.cc:654] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.042510 11892 catalog_manager.cc:5495] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 0 to 1, leader changed from <none> to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:13:24.046155 12311 tablet_bootstrap.cc:492] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:24.046603 12311 ts_tablet_manager.cc:1397] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.061s user 0.008s sys 0.003s
I20260808 20:13:24.047089 12313 tablet_bootstrap.cc:492] T f4918de0ffe94198a66ac1975ddbc693 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:24.047473 12313 ts_tablet_manager.cc:1397] T f4918de0ffe94198a66ac1975ddbc693 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.000s
I20260808 20:13:24.047528 12309 tablet_bootstrap.cc:492] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:24.047955 12309 ts_tablet_manager.cc:1397] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.003s
I20260808 20:13:24.049229 12311 raft_consensus.cc:348] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.049926 12311 raft_consensus.cc:374] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.049819 12313 raft_consensus.cc:348] T f4918de0ffe94198a66ac1975ddbc693 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.050233 12311 raft_consensus.cc:729] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.050501 12313 raft_consensus.cc:374] T f4918de0ffe94198a66ac1975ddbc693 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.050211 12309 raft_consensus.cc:348] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.050798 12313 raft_consensus.cc:729] T f4918de0ffe94198a66ac1975ddbc693 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.050997 12309 raft_consensus.cc:374] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.051311 12309 raft_consensus.cc:729] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.051151 12311 consensus_queue.cc:260] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.051409 12313 consensus_queue.cc:260] T f4918de0ffe94198a66ac1975ddbc693 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.051986 12309 consensus_queue.cc:260] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.053299 12313 ts_tablet_manager.cc:1428] T f4918de0ffe94198a66ac1975ddbc693 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 20:13:24.053860 12309 ts_tablet_manager.cc:1428] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 20:13:24.054082 12313 tablet_bootstrap.cc:492] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:24.054666 12309 tablet_bootstrap.cc:492] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:24.055493 12389 consensus_queue.cc:1035] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:24.059478 12313 tablet_bootstrap.cc:654] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.059818 12309 tablet_bootstrap.cc:654] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.060412 12359 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:24.060842 12359 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.062827 12359 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.068907 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:24.070343 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:13:24.071935 12101 leader_election.cc:341] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
W20260808 20:13:24.073067 12102 leader_election.cc:341] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:24.073489 12102 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:24.078440 12311 ts_tablet_manager.cc:1428] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.032s user 0.004s sys 0.002s
I20260808 20:13:24.079516 12311 tablet_bootstrap.cc:492] T f73cf4b21a754e75b2af936d7402def2 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:24.080813 12359 raft_consensus.cc:2726] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:24.085492 12384 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:24.086805 12384 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.088783 12384 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.090766 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:24.091570 12254 raft_consensus.cc:2379] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 1: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:13:24.093616 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:24.095237 12000 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:24.096408 12384 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
W20260808 20:13:24.102340 12206 consensus_peers.cc:487] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a -> Peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Couldn't send request to peer 78b9263889cd4f1582578a3aa4f644d9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:24.106674 12330 raft_consensus.cc:482] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:24.107223 12330 raft_consensus.cc:504] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.109202 12330 leader_election.cc:288] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.111547 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:24.112221 12150 raft_consensus.cc:2454] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:24.113266 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:24.115623 12254 raft_consensus.cc:2454] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:24.114758 12000 leader_election.cc:302] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:13:24.121819 12330 raft_consensus.cc:2781] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:24.122260 12330 raft_consensus.cc:482] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:24.122581 12330 raft_consensus.cc:3037] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.119450 12359 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:24.124266 12359 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.106735 12311 tablet_bootstrap.cc:654] T f73cf4b21a754e75b2af936d7402def2 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.131906 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:24.132854 12359 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:13:24.133409 12102 leader_election.cc:341] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:24.133966 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:24.134683 12254 raft_consensus.cc:2454] T f4918de0ffe94198a66ac1975ddbc693 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:24.135988 12101 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:24.137816 12360 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:24.138350 12359 raft_consensus.cc:2781] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:24.138298 12360 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.140661 12360 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.141870 12384 raft_consensus.cc:2726] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:24.142918 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:13:24.144464 12102 leader_election.cc:341] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:24.145044 12360 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:24.145484 12360 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.138783 12359 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:24.149362 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:24.148428 12360 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.150449 12330 raft_consensus.cc:504] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.151700 12359 raft_consensus.cc:3037] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
W20260808 20:13:24.154700 12102 leader_election.cc:341] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:24.155910 12330 leader_election.cc:288] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.155880 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:24.157054 12313 tablet_bootstrap.cc:492] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:24.157194 12150 raft_consensus.cc:3037] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.157531 12313 ts_tablet_manager.cc:1397] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.104s user 0.044s sys 0.056s
I20260808 20:13:24.158456 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:24.160024 12313 raft_consensus.cc:348] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.160868 12313 raft_consensus.cc:374] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.159816 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:24.161201 12313 raft_consensus.cc:729] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
W20260808 20:13:24.161441 12101 leader_election.cc:341] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:24.161340 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:24.162474 12254 raft_consensus.cc:3037] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.162021 12313 consensus_queue.cc:260] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.163250 12309 tablet_bootstrap.cc:492] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:24.163681 12309 ts_tablet_manager.cc:1397] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.109s user 0.042s sys 0.060s
I20260808 20:13:24.164376 12313 ts_tablet_manager.cc:1428] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260808 20:13:24.165202 12250 raft_consensus.cc:2454] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:24.161988 12101 leader_election.cc:302] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:24.169771 12360 raft_consensus.cc:2726] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:24.182884 12313 tablet_bootstrap.cc:492] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:24.183964 12311 tablet_bootstrap.cc:492] T f73cf4b21a754e75b2af936d7402def2 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:24.184437 12311 ts_tablet_manager.cc:1397] T f73cf4b21a754e75b2af936d7402def2 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.105s user 0.011s sys 0.003s
I20260808 20:13:24.184198 12309 raft_consensus.cc:348] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.184873 12309 raft_consensus.cc:374] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.185158 12309 raft_consensus.cc:729] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.185854 12309 consensus_queue.cc:260] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.187069 12311 raft_consensus.cc:348] T f73cf4b21a754e75b2af936d7402def2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.187722 12311 raft_consensus.cc:374] T f73cf4b21a754e75b2af936d7402def2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.187861 12309 ts_tablet_manager.cc:1428] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.024s user 0.007s sys 0.015s
I20260808 20:13:24.188002 12311 raft_consensus.cc:729] T f73cf4b21a754e75b2af936d7402def2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.188783 12309 tablet_bootstrap.cc:492] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:24.189103 12313 tablet_bootstrap.cc:654] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.188726 12311 consensus_queue.cc:260] T f73cf4b21a754e75b2af936d7402def2 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.194129 12309 tablet_bootstrap.cc:654] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.195622 12311 ts_tablet_manager.cc:1428] T f73cf4b21a754e75b2af936d7402def2 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260808 20:13:24.196530 12311 tablet_bootstrap.cc:492] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:24.206575 12309 tablet_bootstrap.cc:492] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:24.206992 12309 ts_tablet_manager.cc:1397] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20260808 20:13:24.207056 12254 raft_consensus.cc:2454] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:24.208164 11994 leader_election.cc:302] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:24.209009 12330 raft_consensus.cc:2781] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:24.209447 12330 raft_consensus.cc:686] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:24.210292 12330 consensus_queue.cc:237] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.212445 12311 tablet_bootstrap.cc:654] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.209429 12309 raft_consensus.cc:348] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.215564 12309 raft_consensus.cc:374] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.215817 12309 raft_consensus.cc:729] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.216418 12309 consensus_queue.cc:260] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.217536 12359 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.219903 12359 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.182814 12101 leader_election.cc:302] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:24.221607 12309 ts_tablet_manager.cc:1428] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20260808 20:13:24.222461 12309 tablet_bootstrap.cc:492] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:24.222309 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:13:24.223803 12102 leader_election.cc:341] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:24.224159 11892 catalog_manager.cc:5495] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:24.230068 12309 tablet_bootstrap.cc:654] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.230372 12311 tablet_bootstrap.cc:492] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:24.230849 12311 ts_tablet_manager.cc:1397] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.006s
I20260808 20:13:24.233294 12311 raft_consensus.cc:348] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.234107 12311 raft_consensus.cc:374] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.234373 12311 raft_consensus.cc:729] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.235154 12311 consensus_queue.cc:260] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.237391 12311 ts_tablet_manager.cc:1428] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 20:13:24.238343 12311 tablet_bootstrap.cc:492] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:24.175757 12150 raft_consensus.cc:2454] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:24.241163 12330 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:13:24.241449 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 58726.4us
I20260808 20:13:24.242467 12359 raft_consensus.cc:2781] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:24.242813 12359 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:24.243134 12359 raft_consensus.cc:3037] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.241638 12330 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.246706 12330 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.247476 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:24.247695 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:24.248076 12254 raft_consensus.cc:3037] T f4918de0ffe94198a66ac1975ddbc693 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.248504 12250 raft_consensus.cc:2454] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:24.249011 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:24.249790 11994 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
W20260808 20:13:24.250563 12000 leader_election.cc:341] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:24.250641 12330 raft_consensus.cc:2781] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:24.251048 12330 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:24.251387 12330 raft_consensus.cc:3037] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.258533 12330 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.259397 12254 raft_consensus.cc:2454] T f4918de0ffe94198a66ac1975ddbc693 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:24.261148 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:24.261139 12101 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:24.262462 12360 raft_consensus.cc:2781] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
W20260808 20:13:24.262589 12000 leader_election.cc:341] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:24.262952 12360 raft_consensus.cc:686] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:24.263268 12311 tablet_bootstrap.cc:654] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.263819 12360 consensus_queue.cc:237] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.270495 12309 tablet_bootstrap.cc:492] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:24.270905 12309 ts_tablet_manager.cc:1397] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.049s user 0.015s sys 0.003s
I20260808 20:13:24.273138 12309 raft_consensus.cc:348] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.273788 12309 raft_consensus.cc:374] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.274103 12309 raft_consensus.cc:729] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.275230 12309 consensus_queue.cc:260] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.276467 12330 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.276300 11892 catalog_manager.cc:5495] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:24.277267 12309 ts_tablet_manager.cc:1428] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260808 20:13:24.277120 12359 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.277575 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:24.278162 12254 raft_consensus.cc:3037] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.279102 12309 tablet_bootstrap.cc:492] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:24.279881 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:24.280174 12311 tablet_bootstrap.cc:492] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:24.280478 12047 raft_consensus.cc:3037] T f73cf4b21a754e75b2af936d7402def2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.280601 12311 ts_tablet_manager.cc:1397] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.042s user 0.006s sys 0.007s
I20260808 20:13:24.283113 12311 raft_consensus.cc:348] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.283742 12311 raft_consensus.cc:374] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.283995 12311 raft_consensus.cc:729] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.285353 12309 tablet_bootstrap.cc:654] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.287578 12047 raft_consensus.cc:2454] T f73cf4b21a754e75b2af936d7402def2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:24.290066 12313 tablet_bootstrap.cc:492] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:24.290504 12313 ts_tablet_manager.cc:1397] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.108s user 0.022s sys 0.047s
I20260808 20:13:24.290942 12254 raft_consensus.cc:2454] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:24.292178 12359 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.292121 11994 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:24.293021 12102 leader_election.cc:302] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:13:24.293885 12359 raft_consensus.cc:2781] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:24.294359 12359 raft_consensus.cc:686] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:24.295100 12313 raft_consensus.cc:348] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.295116 12359 consensus_queue.cc:237] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.295981 12313 raft_consensus.cc:374] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.296245 12313 raft_consensus.cc:729] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.297026 12313 consensus_queue.cc:260] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.297503 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:24.298197 12254 raft_consensus.cc:3037] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.299312 12313 ts_tablet_manager.cc:1428] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.009s user 0.004s sys 0.003s
I20260808 20:13:24.299798 12309 tablet_bootstrap.cc:492] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:24.300215 12313 tablet_bootstrap.cc:492] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:24.300264 12309 ts_tablet_manager.cc:1397] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.006s
I20260808 20:13:24.302979 12309 raft_consensus.cc:348] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.303644 12309 raft_consensus.cc:374] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.303948 12309 raft_consensus.cc:729] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.304677 12309 consensus_queue.cc:260] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.307071 12309 ts_tablet_manager.cc:1428] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 20:13:24.307092 12313 tablet_bootstrap.cc:654] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.308161 12309 tablet_bootstrap.cc:492] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:24.309228 12254 raft_consensus.cc:2454] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:24.308179 12311 consensus_queue.cc:260] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.311225 12330 raft_consensus.cc:2781] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:24.312885 12330 raft_consensus.cc:686] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:24.313701 12330 consensus_queue.cc:237] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.314867 12360 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:24.315311 12360 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.317425 12386 raft_consensus.cc:482] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:24.317811 12360 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.318042 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:24.319334 12309 tablet_bootstrap.cc:654] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.320284 12311 ts_tablet_manager.cc:1428] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.039s user 0.004s sys 0.001s
I20260808 20:13:24.321171 12311 tablet_bootstrap.cc:492] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:24.317890 12386 raft_consensus.cc:504] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.328322 12386 leader_election.cc:288] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.329496 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:24.331102 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:24.332042 12254 raft_consensus.cc:2379] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 1: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:13:24.333199 12101 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:24.334128 12360 raft_consensus.cc:3037] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.342113 12360 raft_consensus.cc:2726] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:24.347283 12047 raft_consensus.cc:3037] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.351384 12309 tablet_bootstrap.cc:492] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:24.351817 12309 ts_tablet_manager.cc:1397] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.044s user 0.012s sys 0.006s
I20260808 20:13:24.352553 11892 catalog_manager.cc:5495] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
W20260808 20:13:24.353790 11994 leader_election.cc:341] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:24.354557 12309 raft_consensus.cc:348] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.355240 12309 raft_consensus.cc:374] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.355572 12309 raft_consensus.cc:729] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.355571 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:24.351759 12311 tablet_bootstrap.cc:654] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.356586 12309 consensus_queue.cc:260] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.357542 12150 raft_consensus.cc:2454] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:24.360735 12000 leader_election.cc:302] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:24.361980 12386 raft_consensus.cc:2781] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:24.362314 12386 raft_consensus.cc:482] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:24.362601 12386 raft_consensus.cc:3037] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.366019 12309 ts_tablet_manager.cc:1428] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260808 20:13:24.366884 12309 tablet_bootstrap.cc:492] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:24.365281 11896 catalog_manager.cc:5495] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:24.378348 12309 tablet_bootstrap.cc:654] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.378574 12386 raft_consensus.cc:504] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.381016 12311 tablet_bootstrap.cc:492] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:24.381251 12386 leader_election.cc:288] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.381840 12313 tablet_bootstrap.cc:492] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:24.382275 12313 ts_tablet_manager.cc:1397] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.082s user 0.017s sys 0.013s
I20260808 20:13:24.382687 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:24.383078 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:24.383569 12150 raft_consensus.cc:3037] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.381428 12311 ts_tablet_manager.cc:1397] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.060s user 0.010s sys 0.003s
W20260808 20:13:24.384284 11994 leader_election.cc:341] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:24.386345 12309 tablet_bootstrap.cc:492] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:24.386785 12309 ts_tablet_manager.cc:1397] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.000s
I20260808 20:13:24.387399 12313 raft_consensus.cc:348] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.388041 12313 raft_consensus.cc:374] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.388293 12313 raft_consensus.cc:729] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.388458 12150 raft_consensus.cc:2454] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:24.385993 12311 raft_consensus.cc:348] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.389042 12311 raft_consensus.cc:374] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.389330 12311 raft_consensus.cc:729] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.388886 12313 consensus_queue.cc:260] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.389832 12309 raft_consensus.cc:348] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.390017 12311 consensus_queue.cc:260] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.390589 12309 raft_consensus.cc:374] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.391147 12309 raft_consensus.cc:729] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.391752 12309 consensus_queue.cc:260] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.392185 12311 ts_tablet_manager.cc:1428] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260808 20:13:24.393285 12311 tablet_bootstrap.cc:492] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:24.396951 12313 ts_tablet_manager.cc:1428] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20260808 20:13:24.394937 12309 ts_tablet_manager.cc:1428] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260808 20:13:24.397750 12313 tablet_bootstrap.cc:492] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:24.398079 12309 tablet_bootstrap.cc:492] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:24.397792 12000 leader_election.cc:302] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:24.398993 12386 raft_consensus.cc:2781] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:24.399418 12386 raft_consensus.cc:686] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:24.400199 12386 consensus_queue.cc:237] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.403311 12313 tablet_bootstrap.cc:654] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.411547 12309 tablet_bootstrap.cc:654] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.413677 12359 consensus_queue.cc:1035] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:24.416749 12311 tablet_bootstrap.cc:654] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.418326 11892 catalog_manager.cc:5495] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:24.422843 12313 tablet_bootstrap.cc:492] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:24.423822 12313 ts_tablet_manager.cc:1397] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.000s
I20260808 20:13:24.427963 12309 tablet_bootstrap.cc:492] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:24.428393 12309 ts_tablet_manager.cc:1397] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.005s
I20260808 20:13:24.428381 12313 raft_consensus.cc:348] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.429198 12311 tablet_bootstrap.cc:492] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:24.429190 12313 raft_consensus.cc:374] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.429608 12313 raft_consensus.cc:729] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.429736 12311 ts_tablet_manager.cc:1397] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.037s user 0.014s sys 0.003s
I20260808 20:13:24.431052 12309 raft_consensus.cc:348] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.431151 12313 consensus_queue.cc:260] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.431691 12309 raft_consensus.cc:374] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.431957 12309 raft_consensus.cc:729] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.433166 12313 ts_tablet_manager.cc:1428] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.009s user 0.004s sys 0.002s
I20260808 20:13:24.432878 12309 consensus_queue.cc:260] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.434059 12313 tablet_bootstrap.cc:492] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:24.433862 12386 consensus_queue.cc:1035] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:24.441224 12359 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:24.441458 12313 tablet_bootstrap.cc:654] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.442608 12311 raft_consensus.cc:348] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.443274 12311 raft_consensus.cc:374] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.443526 12311 raft_consensus.cc:729] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.447343 12311 consensus_queue.cc:260] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.462618 12386 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:24.448087 12359 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.463102 12386 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.465395 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:13:24.466656 12101 leader_election.cc:341] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:24.466848 12386 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.448446 12330 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:24.467396 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:24.467741 12330 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:13:24.468508 11994 leader_election.cc:341] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:24.469135 12359 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.470069 12330 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.470974 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:24.471555 12251 raft_consensus.cc:2454] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:24.471875 12311 ts_tablet_manager.cc:1428] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.042s user 0.005s sys 0.000s
I20260808 20:13:24.472483 11994 leader_election.cc:302] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:24.473206 12386 raft_consensus.cc:2781] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:24.473538 12386 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:24.473909 12386 raft_consensus.cc:3037] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.475113 12309 ts_tablet_manager.cc:1428] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.046s user 0.002s sys 0.003s
I20260808 20:13:24.477085 12309 tablet_bootstrap.cc:492] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:24.482410 12311 tablet_bootstrap.cc:492] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:24.480423 12386 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.494298 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:24.494776 12047 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:24.495908 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:24.496488 12150 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:24.497735 12386 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.497870 12313 tablet_bootstrap.cc:492] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:24.498363 12313 ts_tablet_manager.cc:1397] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.064s user 0.031s sys 0.024s
I20260808 20:13:24.500882 12313 raft_consensus.cc:348] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.502023 12313 raft_consensus.cc:374] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.501319 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:24.502333 12313 raft_consensus.cc:729] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.502516 12250 raft_consensus.cc:3037] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.502996 12313 consensus_queue.cc:260] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.504937 12313 ts_tablet_manager.cc:1428] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 20:13:24.505769 12313 tablet_bootstrap.cc:492] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:24.509145 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:24.509791 12150 raft_consensus.cc:2454] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:24.511224 12311 tablet_bootstrap.cc:654] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.512804 12309 tablet_bootstrap.cc:654] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.514025 12313 tablet_bootstrap.cc:654] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.515434 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:24.516016 12150 raft_consensus.cc:3037] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.517685 12250 raft_consensus.cc:2454] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:24.522702 11994 leader_election.cc:302] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:24.523473 12311 tablet_bootstrap.cc:492] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:24.523528 12386 raft_consensus.cc:2781] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:24.523590 12000 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:24.524173 12386 raft_consensus.cc:686] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:24.524724 12384 raft_consensus.cc:2781] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:24.525103 12384 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:24.525481 12384 raft_consensus.cc:3037] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.525007 12386 consensus_queue.cc:237] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.526299 12102 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:24.526229 12437 consensus_queue.cc:1035] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:24.527673 12437 raft_consensus.cc:2781] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:24.528043 12437 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:24.527871 12330 consensus_queue.cc:1035] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:24.528388 12437 raft_consensus.cc:3037] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.524061 12311 ts_tablet_manager.cc:1397] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.009s
I20260808 20:13:24.531852 12311 raft_consensus.cc:348] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.533428 12311 raft_consensus.cc:374] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.533711 12311 raft_consensus.cc:729] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.522720 12150 raft_consensus.cc:2454] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:24.534484 12311 consensus_queue.cc:260] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.536883 12437 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.537273 12311 ts_tablet_manager.cc:1428] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.008s user 0.004s sys 0.001s
I20260808 20:13:24.538745 12311 tablet_bootstrap.cc:492] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:24.540162 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:24.540295 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
W20260808 20:13:24.541764 12101 leader_election.cc:341] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:24.542608 12437 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.544077 12311 tablet_bootstrap.cc:654] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.548031 12309 tablet_bootstrap.cc:492] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:24.548494 12309 ts_tablet_manager.cc:1397] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.072s user 0.009s sys 0.005s
I20260808 20:13:24.549773 12313 tablet_bootstrap.cc:492] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:24.550186 12313 ts_tablet_manager.cc:1397] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.045s user 0.009s sys 0.003s
I20260808 20:13:24.551060 12384 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.552204 12048 raft_consensus.cc:2379] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 1: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:13:24.551857 12309 raft_consensus.cc:348] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.552335 12313 raft_consensus.cc:348] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.553009 12313 raft_consensus.cc:374] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.553205 12313 raft_consensus.cc:729] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.553691 12313 consensus_queue.cc:260] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.552771 12309 raft_consensus.cc:374] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.554031 12102 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:24.554379 12309 raft_consensus.cc:729] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.554914 12437 raft_consensus.cc:2726] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 20:13:24.554986 12309 consensus_queue.cc:260] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:13:24.556500 11966 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 20:13:24.558020 12313 ts_tablet_manager.cc:1428] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260808 20:13:24.558995 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:24.559820 12152 raft_consensus.cc:2379] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 1: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:13:24.559981 12384 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.556648 12309 ts_tablet_manager.cc:1428] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20260808 20:13:24.560938 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:24.561522 12309 tablet_bootstrap.cc:492] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
W20260808 20:13:24.562237 11994 leader_election.cc:341] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:24.563346 12000 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:24.564446 12384 raft_consensus.cc:2726] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 20:13:24.564720 12313 tablet_bootstrap.cc:492] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:24.567018 11897 catalog_manager.cc:5495] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:24.567703 12311 tablet_bootstrap.cc:492] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:24.568789 12321 consensus_queue.cc:1035] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:24.569202 12311 ts_tablet_manager.cc:1397] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.005s
I20260808 20:13:24.571837 12311 raft_consensus.cc:348] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.572443 12311 raft_consensus.cc:374] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.572687 12311 raft_consensus.cc:729] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.573292 12311 consensus_queue.cc:260] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.575124 12311 ts_tablet_manager.cc:1428] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260808 20:13:24.575870 12311 tablet_bootstrap.cc:492] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:24.580008 12309 tablet_bootstrap.cc:654] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.575838 12313 tablet_bootstrap.cc:654] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.589638 12311 tablet_bootstrap.cc:654] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.595818 12389 consensus_queue.cc:1035] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:24.596665 12309 tablet_bootstrap.cc:492] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:24.597096 12309 ts_tablet_manager.cc:1397] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.006s
I20260808 20:13:24.598848 12147 raft_consensus.cc:3037] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.599577 12309 raft_consensus.cc:348] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.600246 12309 raft_consensus.cc:374] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.600498 12309 raft_consensus.cc:729] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.603498 12313 tablet_bootstrap.cc:492] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:24.603920 12313 ts_tablet_manager.cc:1397] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.007s
I20260808 20:13:24.605151 12384 raft_consensus.cc:482] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:24.605589 12384 raft_consensus.cc:504] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.606406 12313 raft_consensus.cc:348] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.607092 12321 raft_consensus.cc:482] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:24.607394 12309 consensus_queue.cc:260] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.607605 12321 raft_consensus.cc:504] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.609543 12309 ts_tablet_manager.cc:1428] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20260808 20:13:24.609937 12321 leader_election.cc:288] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:24.610402 12309 tablet_bootstrap.cc:492] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:24.607046 12313 raft_consensus.cc:374] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.610049 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:24.610836 12313 raft_consensus.cc:729] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.611233 12146 raft_consensus.cc:2454] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:24.611459 12313 consensus_queue.cc:260] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.612993 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:13:24.614777 11994 leader_election.cc:341] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:24.616418 12000 leader_election.cc:302] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:24.617363 12330 raft_consensus.cc:2781] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:24.617290 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:24.617766 12330 raft_consensus.cc:482] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:24.617977 12146 raft_consensus.cc:2454] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 0.
I20260808 20:13:24.618541 12386 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:24.618709 12309 tablet_bootstrap.cc:654] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:24.618127 12330 raft_consensus.cc:3037] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.623291 12313 ts_tablet_manager.cc:1428] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.019s user 0.006s sys 0.001s
I20260808 20:13:24.607613 12384 leader_election.cc:288] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.626187 12313 tablet_bootstrap.cc:492] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:24.624207 12386 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.636384 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:24.638032 12206 leader_election.cc:302] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
W20260808 20:13:24.644824 12205 leader_election.cc:341] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:24.645063 12321 raft_consensus.cc:2781] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:24.645419 12321 raft_consensus.cc:482] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:24.645731 12321 raft_consensus.cc:3037] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.660482 12384 consensus_queue.cc:1035] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260808 20:13:24.668025 12386 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.668568 12386 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:24.670311 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:24.670991 12152 raft_consensus.cc:2454] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:24.672288 12000 leader_election.cc:302] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:13:24.681337 12386 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.683049 12360 consensus_queue.cc:1035] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:24.685922 12386 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.687351 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:24.690717 12386 raft_consensus.cc:2781] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:24.698760 12321 raft_consensus.cc:504] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.697409 12386 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:24.701155 12321 leader_election.cc:288] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:24.701367 12386 raft_consensus.cc:3037] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.701943 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:13:24.703593 12205 leader_election.cc:341] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:24.706712 12321 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:13:24.694770 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 71946us
W20260808 20:13:24.711262 12085 tablet.cc:2369] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 20:13:24.712540 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:24.717197 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:24.718430 12152 raft_consensus.cc:3037] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.713217 12360 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:24.721369 12360 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.711272 12321 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.726250 12321 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:24.750849 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:24.752965 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:24.730612 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:24.763120 12437 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:13:24.751096 12360 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.781204 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:24.787721 12249 raft_consensus.cc:2454] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:24.789548 12101 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:24.782935 12313 tablet_bootstrap.cc:654] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
W20260808 20:13:24.750098 11994 leader_election.cc:341] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:24.782271 12386 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:13:24.793728 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 43686.2us
I20260808 20:13:24.782735 12437 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.772218 12046 raft_consensus.cc:3037] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
W20260808 20:13:24.786876 12205 leader_election.cc:341] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:24.781864 12150 raft_consensus.cc:2454] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 0.
I20260808 20:13:24.763842 12330 raft_consensus.cc:504] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.785496 12387 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:24.796638 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:24.805822 12206 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:24.806443 12330 leader_election.cc:288] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.807075 12152 raft_consensus.cc:2454] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:13:24.807184 12389 raft_consensus.cc:2781] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:24.809731 12389 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:24.810065 12389 raft_consensus.cc:3037] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.811146 12311 tablet_bootstrap.cc:492] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:24.811615 12311 ts_tablet_manager.cc:1397] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.236s user 0.051s sys 0.070s
I20260808 20:13:24.809736 12206 leader_election.cc:302] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:24.814603 12321 raft_consensus.cc:2781] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:24.802310 12360 raft_consensus.cc:2781] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:24.818703 12309 tablet_bootstrap.cc:492] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:24.818629 12360 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:24.819274 12309 ts_tablet_manager.cc:1397] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.209s user 0.008s sys 0.031s
I20260808 20:13:24.819218 12360 raft_consensus.cc:3037] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.812216 12387 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.822562 12311 raft_consensus.cc:348] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.823112 12311 raft_consensus.cc:374] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.823308 12311 raft_consensus.cc:729] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.823778 12311 consensus_queue.cc:260] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.827109 12387 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:24.826937 12319 consensus_queue.cc:1035] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:24.802093 11994 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:24.834556 12311 ts_tablet_manager.cc:1428] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.023s user 0.004s sys 0.002s
I20260808 20:13:24.836372 12437 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:24.838160 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:24.838323 12386 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:13:24.832708 12102 leader_election.cc:341] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
W20260808 20:13:24.839520 12206 leader_election.cc:341] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:24.841171 12455 consensus_queue.cc:1035] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260808 20:13:24.834236 12321 raft_consensus.cc:686] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:24.844736 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:24.843711 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:24.846668 12321 consensus_queue.cc:237] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.850395 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:24.851101 12150 raft_consensus.cc:3037] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.848717 12387 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:24.852209 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:24.852725 12387 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.854141 12102 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:24.854647 12387 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:24.855525 12359 raft_consensus.cc:2726] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:13:24.855827 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:24.852833 12152 raft_consensus.cc:3037] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:24.861284 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:13:24.857436 12206 leader_election.cc:341] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:24.832186 12389 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:13:24.862109 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 62402.9us
I20260808 20:13:24.863541 12437 consensus_queue.cc:1035] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260808 20:13:24.865123 12205 leader_election.cc:341] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:24.865255 12455 consensus_queue.cc:1035] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:24.842661 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:24.865682 12205 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:24.865694 12389 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:24.876825 12311 tablet_bootstrap.cc:492] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:24.883901 12389 raft_consensus.cc:482] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:24.898128 12309 raft_consensus.cc:348] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.898852 12309 raft_consensus.cc:374] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:24.899130 12309 raft_consensus.cc:729] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:24.903841 12309 consensus_queue.cc:260] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:24.956354 12321 raft_consensus.cc:2726] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:24.961869 12437 raft_consensus.cc:482] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:24.979506 12330 raft_consensus.cc:2726] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:24.997012 12387 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:24.935213 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:25.141251 12359 raft_consensus.cc:482] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:24.976651 12152 raft_consensus.cc:2454] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:25.092572 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:25.001510 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:24.904222 12389 raft_consensus.cc:504] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.212157 12437 raft_consensus.cc:504] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.217890 12437 leader_election.cc:288] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:25.218657 12437 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:25.219069 12437 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.043521 12360 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.222512 12311 tablet_bootstrap.cc:654] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:25.213382 12386 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:25.229653 12437 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:25.244122 12313 tablet_bootstrap.cc:492] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:25.211239 12387 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.244552 12313 ts_tablet_manager.cc:1397] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.620s user 0.022s sys 0.041s
I20260808 20:13:25.214123 12359 raft_consensus.cc:504] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.227739 12251 raft_consensus.cc:3037] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:25.228092 12455 consensus_queue.cc:1035] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260808 20:13:25.252068 12360 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:25.252202 12359 leader_election.cc:288] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:25.253116 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:25.253470 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:25.249012 12144 raft_consensus.cc:2379] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 1: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:13:25.253067 12150 raft_consensus.cc:2454] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
W20260808 20:13:25.256661 11994 leader_election.cc:341] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
W20260808 20:13:25.257449 11994 leader_election.cc:341] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
W20260808 20:13:25.258488 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 393378us
I20260808 20:13:25.235145 12386 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.264544 11897 catalog_manager.cc:5495] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 0 to 1, leader changed from <none> to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:13:25.269240 12482 consensus_queue.cc:1035] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20260808 20:13:25.270720 12205 leader_election.cc:341] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
W20260808 20:13:25.290161 12466 log.cc:924] Time spent T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20260808 20:13:25.289094 12389 leader_election.cc:288] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:25.271965 12205 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:25.293473 12389 raft_consensus.cc:2726] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:25.295727 12386 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:25.296411 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:25.298837 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:25.299283 12000 leader_election.cc:302] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:25.303054 12311 tablet_bootstrap.cc:492] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:25.303462 12311 ts_tablet_manager.cc:1397] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.429s user 0.021s sys 0.049s
I20260808 20:13:25.276276 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:25.305902 12472 raft_consensus.cc:2781] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:25.307585 12313 raft_consensus.cc:348] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.308226 12313 raft_consensus.cc:374] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:25.308458 12313 raft_consensus.cc:729] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:25.309029 12313 consensus_queue.cc:260] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.311635 12469 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:25.312104 12469 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.303824 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:25.303930 12045 raft_consensus.cc:3037] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:25.320333 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:25.321494 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:25.322288 12250 raft_consensus.cc:2379] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 1: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:13:25.324153 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:25.310818 12472 raft_consensus.cc:686] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
W20260808 20:13:25.325423 12205 leader_election.cc:341] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:25.325233 12472 consensus_queue.cc:237] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.313462 12387 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:25.328066 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:25.328290 12483 consensus_queue.cc:1035] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
W20260808 20:13:25.313853 12102 leader_election.cc:341] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:25.333271 12309 ts_tablet_manager.cc:1428] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.514s user 0.035s sys 0.043s
I20260808 20:13:25.334880 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:25.335669 12313 ts_tablet_manager.cc:1428] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.090s user 0.005s sys 0.000s
I20260808 20:13:25.336519 12313 tablet_bootstrap.cc:492] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:25.338008 12311 raft_consensus.cc:348] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.339332 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:25.340137 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:25.330873 12330 consensus_queue.cc:1035] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260808 20:13:25.337486 12000 leader_election.cc:302] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:25.341892 12206 leader_election.cc:302] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:13:25.345453 12101 leader_election.cc:341] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:25.347051 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:25.349653 12311 raft_consensus.cc:374] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260808 20:13:25.333602 12309 ts_tablet_manager.cc:1460] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9: Tablet startup took 513ms
W20260808 20:13:25.356920 12463 log.cc:924] Time spent T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.106s user 0.000s sys 0.001s
I20260808 20:13:25.357149 12250 raft_consensus.cc:2379] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 1: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:13:25.349251 12144 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 0.
I20260808 20:13:25.358047 12311 raft_consensus.cc:729] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:25.359439 12311 consensus_queue.cc:260] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.337054 12475 consensus_queue.cc:1035] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.106s
I20260808 20:13:25.365867 12311 ts_tablet_manager.cc:1428] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.062s user 0.004s sys 0.003s
I20260808 20:13:25.366494 12311 tablet_bootstrap.cc:492] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:25.358717 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:13:25.358305 12309 ts_tablet_manager.cc:1462] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9: Trace:
0808 20:13:22.255870 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:13:22.458992 (+203122us) service_pool.cc:227] Handling call
0808 20:13:22.460592 (+ 1600us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0808 20:13:22.460679 (+ 87us) ts_tablet_manager.cc:676] Creating new metadata...
0808 20:13:22.556934 (+ 96255us) tablet_metadata.cc:641] Metadata flushed
0808 20:13:22.631397 (+ 74463us) tablet_replica.cc:205] Creating consensus instance
0808 20:13:22.638999 (+ 7602us) inbound_call.cc:173] Queueing success response
0808 20:13:24.610121 (+1971122us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0808 20:13:24.819666 (+209545us) ts_tablet_manager.cc:1429] Starting tablet replica
0808 20:13:24.892181 (+ 72515us) tablet_replica.cc:259] Starting instrumentation
0808 20:13:24.895260 (+ 3079us) tablet_replica.cc:278] Starting consensus
Metrics: {"raft.queue_time_us":26098,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":81,"tablet-open.queue_time_us":1971290,"thread_start_us":136249,"threads_started":1}
W20260808 20:13:25.358939 12102 leader_election.cc:341] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:25.370028 12309 tablet_bootstrap.cc:492] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:25.362988 12483 raft_consensus.cc:2781] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:25.368098 12468 raft_consensus.cc:2726] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:25.369417 12469 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:13:25.372790 12101 leader_election.cc:341] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
W20260808 20:13:25.377905 12205 leader_election.cc:341] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:25.373772 12483 raft_consensus.cc:686] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:25.379268 12330 consensus_queue.cc:1035] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.026s
I20260808 20:13:25.405639 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:25.406353 12311 tablet_bootstrap.cc:654] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:25.400027 12101 leader_election.cc:302] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:25.409571 12488 raft_consensus.cc:2726] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 20:13:25.410099 12101 leader_election.cc:341] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:25.414292 12102 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:25.419020 12482 raft_consensus.cc:2726] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:25.396534 12483 consensus_queue.cc:237] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:13:25.417809 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 71265.5us
W20260808 20:13:25.420994 12102 leader_election.cc:341] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:25.421648 12102 leader_election.cc:302] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:25.411428 12206 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:25.412637 12330 raft_consensus.cc:482] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:25.429445 12330 raft_consensus.cc:504] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.431960 12046 raft_consensus.cc:3037] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:25.432317 12330 leader_election.cc:288] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:25.439186 12488 raft_consensus.cc:2726] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:25.444653 12469 raft_consensus.cc:2781] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:25.445055 12469 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:25.445348 12469 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:25.447769 12468 consensus_queue.cc:1035] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:25.449074 12045 raft_consensus.cc:2454] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:13:25.457237 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:13:25.460494 12206 leader_election.cc:341] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:25.467933 12206 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:25.487321 12387 raft_consensus.cc:2726] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:25.456121 12102 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:25.491334 12205 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:25.492322 12488 raft_consensus.cc:2726] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 20:13:25.492596 12468 raft_consensus.cc:2781] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:25.493120 12468 raft_consensus.cc:686] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:25.493953 12468 consensus_queue.cc:237] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.510177 12313 tablet_bootstrap.cc:654] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
W20260808 20:13:25.523615 12492 log.cc:924] Time spent T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
W20260808 20:13:25.511478 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 51533.4us
I20260808 20:13:25.541386 12309 tablet_bootstrap.cc:654] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:25.543069 12311 tablet_bootstrap.cc:492] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:25.543406 12311 ts_tablet_manager.cc:1397] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.177s user 0.034s sys 0.033s
I20260808 20:13:25.544000 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:25.545557 12152 raft_consensus.cc:2379] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 1: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:13:25.548271 12469 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.549264 12387 consensus_queue.cc:1035] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:25.552016 12469 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:25.553858 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:25.557852 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:25.558586 12146 raft_consensus.cc:2379] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 1: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:13:25.558874 12000 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:25.563513 12386 raft_consensus.cc:2726] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 20:13:25.590497 12205 leader_election.cc:341] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:25.590612 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:25.591300 12142 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:25.593483 12311 raft_consensus.cc:348] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.593964 12311 raft_consensus.cc:374] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:25.594147 12311 raft_consensus.cc:729] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:25.594616 12311 consensus_queue.cc:260] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.612702 12000 leader_election.cc:302] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:25.614341 12475 raft_consensus.cc:2726] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 20:13:25.618652 11966 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 20:13:25.622617 12313 tablet_bootstrap.cc:492] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:25.623142 12313 ts_tablet_manager.cc:1397] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.287s user 0.010s sys 0.007s
I20260808 20:13:25.625175 12311 ts_tablet_manager.cc:1428] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.082s user 0.004s sys 0.002s
I20260808 20:13:25.625849 12311 tablet_bootstrap.cc:492] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:25.628448 12313 raft_consensus.cc:348] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.629208 12313 raft_consensus.cc:374] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:25.629508 12313 raft_consensus.cc:729] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:25.635826 12387 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:25.636512 12142 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:13:25.636319 12387 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.638448 12387 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:25.630224 12313 consensus_queue.cc:260] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.643314 12313 ts_tablet_manager.cc:1428] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20260808 20:13:25.641547 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:25.644769 12313 tablet_bootstrap.cc:492] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:25.644976 12043 raft_consensus.cc:2379] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 1: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:13:25.646878 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:25.644834 12206 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:25.647701 12142 raft_consensus.cc:2379] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 1: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:13:25.647946 12469 raft_consensus.cc:2781] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:25.648398 12469 raft_consensus.cc:686] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:25.649272 12469 consensus_queue.cc:237] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.661509 12206 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:25.662395 12387 raft_consensus.cc:3037] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:25.665716 12311 tablet_bootstrap.cc:654] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:25.667279 12309 tablet_bootstrap.cc:492] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:25.667869 12309 ts_tablet_manager.cc:1397] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.298s user 0.010s sys 0.006s
I20260808 20:13:25.670161 12387 raft_consensus.cc:2726] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:25.670238 12309 raft_consensus.cc:348] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.670929 12309 raft_consensus.cc:374] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:25.671214 12309 raft_consensus.cc:729] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:25.671808 12309 consensus_queue.cc:260] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.674213 12309 ts_tablet_manager.cc:1428] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260808 20:13:25.671353 11896 catalog_manager.cc:5495] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 0 to 1, leader changed from <none> to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:13:25.675014 12309 tablet_bootstrap.cc:492] T 0be6c610fe68472b8bcb6232b862dbc1 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:25.675256 12488 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:25.675691 12488 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.678309 12488 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:25.679517 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:25.680024 12250 raft_consensus.cc:2454] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:25.681105 12101 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:25.681257 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:25.682001 12488 raft_consensus.cc:2781] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:25.682387 12488 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:13:25.682644 12102 leader_election.cc:341] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:25.682698 12488 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:25.688316 12311 tablet_bootstrap.cc:492] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:25.688753 12311 ts_tablet_manager.cc:1397] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.005s
I20260808 20:13:25.691085 12311 raft_consensus.cc:348] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.691588 12309 tablet_bootstrap.cc:654] T 0be6c610fe68472b8bcb6232b862dbc1 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:25.691789 12311 raft_consensus.cc:374] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:25.692044 12311 raft_consensus.cc:729] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:25.692725 12311 consensus_queue.cc:260] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.691250 12488 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.693408 11892 catalog_manager.cc:5495] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:25.694892 12311 ts_tablet_manager.cc:1428] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260808 20:13:25.695755 12313 tablet_bootstrap.cc:654] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:25.701565 12311 tablet_bootstrap.cc:492] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:25.699100 12309 tablet_bootstrap.cc:492] T 0be6c610fe68472b8bcb6232b862dbc1 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:25.703787 12309 ts_tablet_manager.cc:1397] T 0be6c610fe68472b8bcb6232b862dbc1 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.003s
I20260808 20:13:25.706121 12488 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:25.707291 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:25.707851 12311 tablet_bootstrap.cc:654] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:25.708180 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:25.708384 12309 raft_consensus.cc:348] T 0be6c610fe68472b8bcb6232b862dbc1 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:13:25.709470 12102 leader_election.cc:341] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:25.707854 12250 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:25.710153 12309 raft_consensus.cc:374] T 0be6c610fe68472b8bcb6232b862dbc1 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:25.710752 12309 raft_consensus.cc:729] T 0be6c610fe68472b8bcb6232b862dbc1 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:25.711192 11892 catalog_manager.cc:5495] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:25.711365 12309 consensus_queue.cc:260] T 0be6c610fe68472b8bcb6232b862dbc1 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.714864 12309 ts_tablet_manager.cc:1428] T 0be6c610fe68472b8bcb6232b862dbc1 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20260808 20:13:25.715762 12309 tablet_bootstrap.cc:492] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:25.722101 12309 tablet_bootstrap.cc:654] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:25.729354 11896 catalog_manager.cc:5495] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 0 to 1, leader changed from <none> to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:13:25.740739 12250 raft_consensus.cc:2454] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:25.742020 12101 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:25.742924 12488 raft_consensus.cc:2781] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:25.743425 12488 raft_consensus.cc:686] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:25.744303 12488 consensus_queue.cc:237] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.757151 12313 tablet_bootstrap.cc:492] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:25.757550 12313 ts_tablet_manager.cc:1397] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.113s user 0.029s sys 0.016s
I20260808 20:13:25.759178 12313 raft_consensus.cc:348] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.759781 12313 raft_consensus.cc:374] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:25.760076 12313 raft_consensus.cc:729] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:25.760614 12313 consensus_queue.cc:260] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.762445 12313 ts_tablet_manager.cc:1428] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260808 20:13:25.762521 11892 catalog_manager.cc:5495] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:25.763550 12313 tablet_bootstrap.cc:492] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:25.785107 12313 tablet_bootstrap.cc:654] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:25.785959 12309 tablet_bootstrap.cc:492] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:25.786435 12309 ts_tablet_manager.cc:1397] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.071s user 0.012s sys 0.008s
I20260808 20:13:25.786531 12311 tablet_bootstrap.cc:492] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:25.786989 12311 ts_tablet_manager.cc:1397] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.015s
I20260808 20:13:25.788980 12309 raft_consensus.cc:348] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.789234 12311 raft_consensus.cc:348] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.789714 12309 raft_consensus.cc:374] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:25.789911 12311 raft_consensus.cc:374] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:25.790012 12309 raft_consensus.cc:729] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:25.790226 12311 raft_consensus.cc:729] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:25.790802 12309 consensus_queue.cc:260] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.791697 12311 consensus_queue.cc:260] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.793174 12309 ts_tablet_manager.cc:1428] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 20:13:25.793748 12311 ts_tablet_manager.cc:1428] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260808 20:13:25.794509 12309 tablet_bootstrap.cc:492] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:25.795111 12311 tablet_bootstrap.cc:492] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
W20260808 20:13:25.805594 11994 consensus_peers.cc:487] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 -> Peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Couldn't send request to peer b9d94f35ea15421aa1d721201c782d8a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:25.809737 12309 tablet_bootstrap.cc:654] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:25.810293 12311 tablet_bootstrap.cc:654] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:25.814759 12313 tablet_bootstrap.cc:492] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:25.815160 12313 ts_tablet_manager.cc:1397] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.052s user 0.014s sys 0.001s
I20260808 20:13:25.817271 12472 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:25.817404 12313 raft_consensus.cc:348] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.818094 12313 raft_consensus.cc:374] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:25.817754 12472 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.818336 12313 raft_consensus.cc:729] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:25.819010 12313 consensus_queue.cc:260] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.819697 12472 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:25.821220 12311 tablet_bootstrap.cc:492] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:25.821686 12311 ts_tablet_manager.cc:1397] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.004s
I20260808 20:13:25.821389 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:13:25.822913 12000 leader_election.cc:341] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:25.823339 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:25.824554 12313 ts_tablet_manager.cc:1428] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260808 20:13:25.824308 12311 raft_consensus.cc:348] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.825147 12311 raft_consensus.cc:374] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:25.825512 12313 tablet_bootstrap.cc:492] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:25.825469 12311 raft_consensus.cc:729] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:25.826249 12311 consensus_queue.cc:260] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.828750 12311 ts_tablet_manager.cc:1428] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260808 20:13:25.828838 11994 leader_election.cc:341] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:25.829676 11994 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:25.830945 12472 raft_consensus.cc:2726] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:25.831904 12311 tablet_bootstrap.cc:492] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:25.835603 12313 tablet_bootstrap.cc:654] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:25.842697 12311 tablet_bootstrap.cc:654] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:25.845347 12309 tablet_bootstrap.cc:492] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:25.847436 12309 ts_tablet_manager.cc:1397] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.053s user 0.011s sys 0.029s
I20260808 20:13:25.850353 12313 tablet_bootstrap.cc:492] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:25.850795 12313 ts_tablet_manager.cc:1397] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.002s
I20260808 20:13:25.852698 12488 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:25.853147 12488 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.853670 12313 raft_consensus.cc:348] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.854264 12313 raft_consensus.cc:374] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:25.854502 12313 raft_consensus.cc:729] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:25.855012 12488 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:25.856020 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:25.856694 12251 raft_consensus.cc:2454] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:25.857877 12101 leader_election.cc:302] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:25.858619 12488 raft_consensus.cc:2781] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:25.858973 12488 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:25.859266 12488 raft_consensus.cc:3037] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:25.859493 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:25.859383 12313 consensus_queue.cc:260] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:13:25.860850 12102 leader_election.cc:341] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:25.862993 12313 ts_tablet_manager.cc:1428] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20260808 20:13:25.863799 12313 tablet_bootstrap.cc:492] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:25.865723 12488 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.868770 12313 tablet_bootstrap.cc:654] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:25.868659 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:13:25.869947 12102 leader_election.cc:341] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:25.859247 12309 raft_consensus.cc:348] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.870596 12309 raft_consensus.cc:374] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:25.870904 12309 raft_consensus.cc:729] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:25.871541 12309 consensus_queue.cc:260] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.872303 12311 tablet_bootstrap.cc:492] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:25.873972 12309 ts_tablet_manager.cc:1428] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.026s user 0.005s sys 0.003s
I20260808 20:13:25.874475 12488 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:25.875556 12309 tablet_bootstrap.cc:492] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:25.875671 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:25.876237 12251 raft_consensus.cc:3037] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:25.876413 12311 ts_tablet_manager.cc:1397] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.000s
I20260808 20:13:25.878847 12311 raft_consensus.cc:348] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.879444 12311 raft_consensus.cc:374] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:25.879703 12311 raft_consensus.cc:729] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:25.880300 12311 consensus_queue.cc:260] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.881076 12250 raft_consensus.cc:3037] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:25.881809 12309 tablet_bootstrap.cc:654] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:25.882280 12311 ts_tablet_manager.cc:1428] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260808 20:13:25.883157 12311 tablet_bootstrap.cc:492] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:25.883627 12251 raft_consensus.cc:2454] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:25.884603 12101 leader_election.cc:302] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:25.885596 12482 raft_consensus.cc:2781] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:25.886058 12482 raft_consensus.cc:686] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:25.886806 12482 consensus_queue.cc:237] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.891186 12472 consensus_queue.cc:1035] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:25.895159 12311 tablet_bootstrap.cc:654] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:25.906965 12313 tablet_bootstrap.cc:492] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:25.907371 12313 ts_tablet_manager.cc:1397] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.044s user 0.019s sys 0.008s
I20260808 20:13:25.909667 12313 raft_consensus.cc:348] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.910315 12313 raft_consensus.cc:374] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:25.910596 12313 raft_consensus.cc:729] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:25.911574 12313 consensus_queue.cc:260] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.913753 12313 ts_tablet_manager.cc:1428] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260808 20:13:25.914552 12313 tablet_bootstrap.cc:492] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:25.914968 11892 catalog_manager.cc:5495] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:25.919746 12313 tablet_bootstrap.cc:654] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:25.920135 12311 tablet_bootstrap.cc:492] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:25.920647 12311 ts_tablet_manager.cc:1397] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.002s
I20260808 20:13:25.924569 12311 raft_consensus.cc:348] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.925290 12311 raft_consensus.cc:374] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:25.925577 12311 raft_consensus.cc:729] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:25.926043 12309 tablet_bootstrap.cc:492] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:25.926486 12309 ts_tablet_manager.cc:1397] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.051s user 0.018s sys 0.010s
I20260808 20:13:25.926354 12311 consensus_queue.cc:260] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.928795 12309 raft_consensus.cc:348] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.929467 12309 raft_consensus.cc:374] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:25.932761 12309 raft_consensus.cc:729] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:25.932554 12387 consensus_queue.cc:1035] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:25.933446 12309 consensus_queue.cc:260] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.935212 12311 ts_tablet_manager.cc:1428] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260808 20:13:25.936051 12311 tablet_bootstrap.cc:492] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:25.941246 12311 tablet_bootstrap.cc:654] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:25.943886 12309 ts_tablet_manager.cc:1428] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260808 20:13:25.944756 12309 tablet_bootstrap.cc:492] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:25.950986 12309 tablet_bootstrap.cc:654] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:25.952317 12311 tablet_bootstrap.cc:492] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:25.952765 12311 ts_tablet_manager.cc:1397] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.012s
I20260808 20:13:25.955192 12311 raft_consensus.cc:348] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.956023 12311 raft_consensus.cc:374] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:25.956348 12311 raft_consensus.cc:729] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:25.957043 12311 consensus_queue.cc:260] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.962952 12311 ts_tablet_manager.cc:1428] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.010s user 0.002s sys 0.003s
I20260808 20:13:25.963912 12311 tablet_bootstrap.cc:492] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:25.969336 12311 tablet_bootstrap.cc:654] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:25.971951 12309 tablet_bootstrap.cc:492] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:25.972424 12309 ts_tablet_manager.cc:1397] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.028s user 0.018s sys 0.006s
I20260808 20:13:25.975476 12309 raft_consensus.cc:348] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.976253 12309 raft_consensus.cc:374] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:25.976567 12309 raft_consensus.cc:729] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:25.977310 12309 consensus_queue.cc:260] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.979702 12309 ts_tablet_manager.cc:1428] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260808 20:13:25.980612 12309 tablet_bootstrap.cc:492] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:25.986423 12313 tablet_bootstrap.cc:492] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:25.986881 12313 ts_tablet_manager.cc:1397] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.072s user 0.024s sys 0.035s
I20260808 20:13:25.990561 12472 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:25.990825 12386 consensus_queue.cc:1035] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:25.991014 12472 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:25.993607 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:25.994210 12142 raft_consensus.cc:2454] T 0be6c610fe68472b8bcb6232b862dbc1 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:25.994972 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:25.995257 12000 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:13:25.995585 12250 raft_consensus.cc:2454] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:25.997020 12483 raft_consensus.cc:2781] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:25.997795 12483 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:25.992842 12472 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:25.998319 12483 raft_consensus.cc:3037] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:26.003417 12313 raft_consensus.cc:348] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.004076 12313 raft_consensus.cc:374] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:26.005923 12313 raft_consensus.cc:729] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:26.007236 12309 tablet_bootstrap.cc:654] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:26.007017 12472 consensus_queue.cc:1035] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:26.009037 12313 consensus_queue.cc:260] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.012995 12389 consensus_queue.cc:1035] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:26.026156 12389 raft_consensus.cc:482] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:26.026613 12389 raft_consensus.cc:504] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.029595 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:26.030221 12043 raft_consensus.cc:2454] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 0.
I20260808 20:13:26.031553 12205 leader_election.cc:302] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:26.032385 12469 raft_consensus.cc:2781] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:26.032724 12469 raft_consensus.cc:482] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:26.033062 12469 raft_consensus.cc:3037] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:26.046537 12313 ts_tablet_manager.cc:1428] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.059s user 0.010s sys 0.011s
I20260808 20:13:26.046840 12483 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.047904 12313 tablet_bootstrap.cc:492] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:26.050058 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:26.050726 12250 raft_consensus.cc:3037] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:26.050848 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:26.051460 12144 raft_consensus.cc:3037] T 0be6c610fe68472b8bcb6232b862dbc1 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:26.053205 12313 tablet_bootstrap.cc:654] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:26.053423 12386 raft_consensus.cc:482] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:26.053850 12386 raft_consensus.cc:504] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.055666 12386 leader_election.cc:288] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:26.057418 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:26.058055 12251 raft_consensus.cc:2454] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:26.059475 11994 leader_election.cc:302] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:26.065310 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:26.065930 12152 raft_consensus.cc:2454] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:26.070051 12389 leader_election.cc:288] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:26.069228 12386 raft_consensus.cc:2781] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:26.070989 12386 raft_consensus.cc:482] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:26.071346 12386 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:26.078989 12250 raft_consensus.cc:2454] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:26.080596 12483 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:26.081782 11994 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:26.081970 12144 raft_consensus.cc:2454] T 0be6c610fe68472b8bcb6232b862dbc1 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:26.082885 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:26.085475 12152 raft_consensus.cc:2454] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 0.
I20260808 20:13:26.088209 12330 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:26.088665 12330 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.091656 12330 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:26.084884 12483 raft_consensus.cc:2781] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:26.098073 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:26.100589 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:26.101485 12152 raft_consensus.cc:2379] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 1: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:13:26.102792 12000 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:26.103775 12475 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:26.104324 12475 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.106630 12475 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:26.107128 12475 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:26.110599 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:26.111322 12251 raft_consensus.cc:2379] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 1: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:13:26.111038 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:26.113270 12311 tablet_bootstrap.cc:492] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:26.113622 12311 ts_tablet_manager.cc:1397] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.150s user 0.019s sys 0.015s
I20260808 20:13:26.115417 12483 raft_consensus.cc:686] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:26.118618 12311 raft_consensus.cc:348] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.123706 12313 tablet_bootstrap.cc:492] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:26.123775 12311 raft_consensus.cc:374] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:26.124202 12313 ts_tablet_manager.cc:1397] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.076s user 0.010s sys 0.001s
I20260808 20:13:26.124361 12311 raft_consensus.cc:729] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:26.124986 12311 consensus_queue.cc:260] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.125974 12000 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:26.126565 12311 ts_tablet_manager.cc:1428] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.013s user 0.006s sys 0.003s
I20260808 20:13:26.127275 12472 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:26.128768 12313 raft_consensus.cc:348] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.129493 12313 raft_consensus.cc:374] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:26.129767 12313 raft_consensus.cc:729] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:26.130381 12313 consensus_queue.cc:260] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.133334 12311 tablet_bootstrap.cc:492] T 4e393e90875847809e1ef94179c5b4ef P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:26.124549 12483 consensus_queue.cc:237] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.135217 12321 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:26.135699 12321 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.141250 12313 ts_tablet_manager.cc:1428] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.017s user 0.007s sys 0.001s
I20260808 20:13:26.146001 12313 tablet_bootstrap.cc:492] T a383bcab83014009b5698c63cdac298b P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:26.156981 12321 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:26.149227 12386 raft_consensus.cc:504] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.184314 12387 raft_consensus.cc:482] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:26.184799 12387 raft_consensus.cc:504] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.210559 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:13:26.212507 12205 leader_election.cc:341] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:26.213606 12475 raft_consensus.cc:2726] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:26.219682 12472 raft_consensus.cc:2726] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:26.290606 12470 consensus_queue.cc:1035] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:26.309291 12469 raft_consensus.cc:504] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.285213 12472 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:13:26.280311 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 152619us
I20260808 20:13:26.394801 12469 leader_election.cc:288] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:26.416647 12309 tablet_bootstrap.cc:492] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:26.417188 12309 ts_tablet_manager.cc:1397] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.437s user 0.039s sys 0.107s
I20260808 20:13:26.355226 12472 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.420229 12483 raft_consensus.cc:482] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:26.419458 12488 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:13:26.421357 12562 log.cc:924] Time spent T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.271s user 0.012s sys 0.000s
I20260808 20:13:26.430424 12386 leader_election.cc:288] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:26.432583 12483 raft_consensus.cc:504] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.432848 12488 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.443701 12482 consensus_queue.cc:1035] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
W20260808 20:13:26.448169 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 153837us
I20260808 20:13:26.457962 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:26.462278 12146 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:26.459468 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:26.464540 12311 tablet_bootstrap.cc:654] T 4e393e90875847809e1ef94179c5b4ef P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:26.460414 12313 tablet_bootstrap.cc:654] T a383bcab83014009b5698c63cdac298b P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:26.468043 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:26.468660 12046 raft_consensus.cc:3037] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:26.469185 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:26.472021 12150 raft_consensus.cc:2454] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:26.459949 12472 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:26.464574 12387 leader_election.cc:288] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:26.456353 12573 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:26.471498 12389 raft_consensus.cc:482] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:26.481232 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:26.490108 12251 raft_consensus.cc:3037] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:26.484488 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:26.487826 12573 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.488600 12389 raft_consensus.cc:504] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.493954 12359 consensus_queue.cc:1035] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:26.494462 12146 raft_consensus.cc:2454] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:26.489403 12152 raft_consensus.cc:3037] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:26.495466 12483 leader_election.cc:288] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:26.498138 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:26.499593 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:26.496798 12046 raft_consensus.cc:2454] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:13:26.506841 12309 raft_consensus.cc:348] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.507309 12309 raft_consensus.cc:374] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:26.507507 12309 raft_consensus.cc:729] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:26.506614 12250 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:26.508844 12000 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:13:26.511246 12567 raft_consensus.cc:2781] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:26.513612 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:26.517719 12321 consensus_queue.cc:1035] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:26.518388 12573 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:26.519901 12488 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:26.520335 12152 raft_consensus.cc:2454] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:13:26.512113 12309 consensus_queue.cc:260] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.518355 12576 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:26.511595 12567 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:26.520504 12206 leader_election.cc:302] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:26.524062 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:26.528813 12205 leader_election.cc:302] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:26.528879 12146 raft_consensus.cc:2454] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 0.
I20260808 20:13:26.529646 12321 raft_consensus.cc:2781] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:26.530086 12321 raft_consensus.cc:686] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:26.530253 12309 ts_tablet_manager.cc:1428] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.113s user 0.003s sys 0.003s
I20260808 20:13:26.530983 12389 leader_election.cc:288] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:26.530857 12321 consensus_queue.cc:237] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:13:26.532318 11994 leader_election.cc:341] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:26.532372 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:26.531108 12309 tablet_bootstrap.cc:492] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:26.534699 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:26.537171 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:26.538025 12250 raft_consensus.cc:2454] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
W20260808 20:13:26.538535 12205 leader_election.cc:341] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:26.538913 12000 leader_election.cc:302] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:13:26.539362 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:26.524035 12576 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:13:26.544271 12102 consensus_peers.cc:487] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 -> Peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Couldn't send request to peer 4cd46f13bceb4c3c9ef71e755e17b995. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:26.544409 12483 raft_consensus.cc:2781] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election won for term 1
W20260808 20:13:26.544925 12102 leader_election.cc:341] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:26.540027 12047 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
W20260808 20:13:26.546766 12000 leader_election.cc:341] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:26.525738 12567 raft_consensus.cc:3037] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:26.545107 12483 raft_consensus.cc:686] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:26.553861 12470 raft_consensus.cc:2726] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:26.561656 12311 tablet_bootstrap.cc:492] T 4e393e90875847809e1ef94179c5b4ef P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:26.562130 12311 ts_tablet_manager.cc:1397] T 4e393e90875847809e1ef94179c5b4ef P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.429s user 0.100s sys 0.169s
I20260808 20:13:26.570166 12206 leader_election.cc:302] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:26.571012 12469 raft_consensus.cc:2781] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:26.571352 12469 raft_consensus.cc:482] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:26.571662 12469 raft_consensus.cc:3037] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:26.573398 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:26.578665 12578 consensus_queue.cc:1035] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:26.578341 12571 raft_consensus.cc:482] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:26.580190 12571 raft_consensus.cc:504] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.582023 12571 leader_election.cc:288] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:26.582742 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:26.584313 12483 consensus_queue.cc:237] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.592373 12309 tablet_bootstrap.cc:654] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
W20260808 20:13:26.596603 11994 leader_election.cc:341] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:26.597082 11994 leader_election.cc:302] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:26.597268 12313 tablet_bootstrap.cc:492] T a383bcab83014009b5698c63cdac298b P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:26.597822 12313 ts_tablet_manager.cc:1397] T a383bcab83014009b5698c63cdac298b P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.456s user 0.007s sys 0.012s
I20260808 20:13:26.597796 12472 raft_consensus.cc:2726] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:26.566179 12102 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:13:26.602514 12469 raft_consensus.cc:504] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:13:26.604267 12101 leader_election.cc:341] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:26.604732 12313 raft_consensus.cc:348] T a383bcab83014009b5698c63cdac298b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.605427 12313 raft_consensus.cc:374] T a383bcab83014009b5698c63cdac298b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:26.605679 12313 raft_consensus.cc:729] T a383bcab83014009b5698c63cdac298b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:26.605830 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:26.606168 12469 leader_election.cc:288] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:26.606456 12142 raft_consensus.cc:2454] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 0.
I20260808 20:13:26.607477 12578 raft_consensus.cc:2781] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:26.607532 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:26.607952 12578 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:26.597313 12576 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:26.608317 12578 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:26.608477 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:26.606276 12313 consensus_queue.cc:260] T a383bcab83014009b5698c63cdac298b P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.610257 12313 ts_tablet_manager.cc:1428] T a383bcab83014009b5698c63cdac298b P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260808 20:13:26.610073 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:26.613034 12313 tablet_bootstrap.cc:492] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:26.614009 12311 raft_consensus.cc:348] T 4e393e90875847809e1ef94179c5b4ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:13:26.614710 12102 leader_election.cc:341] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:26.615073 12311 raft_consensus.cc:374] T 4e393e90875847809e1ef94179c5b4ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:26.615413 12311 raft_consensus.cc:729] T 4e393e90875847809e1ef94179c5b4ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:26.614009 12045 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:26.616225 12311 consensus_queue.cc:260] T 4e393e90875847809e1ef94179c5b4ef P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.623378 12311 ts_tablet_manager.cc:1428] T 4e393e90875847809e1ef94179c5b4ef P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.061s user 0.007s sys 0.000s
I20260808 20:13:26.624311 12311 tablet_bootstrap.cc:492] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:26.627734 12567 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.644780 12102 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:13:26.650732 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:26.651358 12142 raft_consensus.cc:3037] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:26.665729 12576 raft_consensus.cc:2781] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:13:26.697161 12043 raft_consensus.cc:3037] T 4e393e90875847809e1ef94179c5b4ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:26.717746 12206 leader_election.cc:302] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:26.718745 12469 raft_consensus.cc:2781] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:26.732129 12469 raft_consensus.cc:482] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:26.733996 12469 raft_consensus.cc:3037] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:26.707624 12576 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:26.755476 12567 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:26.755781 12576 raft_consensus.cc:3037] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:26.756307 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:26.757296 12152 raft_consensus.cc:3037] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:26.760346 12309 tablet_bootstrap.cc:492] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:26.760783 12309 ts_tablet_manager.cc:1397] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.230s user 0.029s sys 0.029s
I20260808 20:13:26.761863 11897 catalog_manager.cc:5495] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 0 to 1, leader changed from <none> to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
W20260808 20:13:26.765097 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 186358us
I20260808 20:13:26.768399 12472 consensus_queue.cc:1035] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 20:13:26.770797 12101 leader_election.cc:341] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:26.771202 12101 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:26.771903 12579 raft_consensus.cc:2726] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 20:13:26.764973 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 150595us
I20260808 20:13:26.778478 12152 raft_consensus.cc:2454] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:13:26.780628 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:26.780705 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:26.784273 12309 raft_consensus.cc:348] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.786144 12251 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:26.786123 12309 raft_consensus.cc:374] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:26.787038 12309 raft_consensus.cc:729] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
W20260808 20:13:26.761613 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 66181.6us
I20260808 20:13:26.817299 12311 tablet_bootstrap.cc:654] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:26.823134 12309 consensus_queue.cc:260] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.829766 12043 raft_consensus.cc:2454] T 4e393e90875847809e1ef94179c5b4ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:13:26.830657 12205 leader_election.cc:302] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:26.831354 12309 ts_tablet_manager.cc:1428] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.070s user 0.004s sys 0.002s
I20260808 20:13:26.831568 12321 raft_consensus.cc:2781] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:26.831995 12321 raft_consensus.cc:686] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:26.832219 12309 tablet_bootstrap.cc:492] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:26.832803 12321 consensus_queue.cc:237] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.819618 12578 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.839346 12576 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.841823 12578 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:26.836400 11894 catalog_manager.cc:5495] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:26.843425 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:26.844125 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:26.846364 12043 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:26.844506 12142 raft_consensus.cc:2454] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:26.847378 12313 tablet_bootstrap.cc:654] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:26.847095 12469 raft_consensus.cc:504] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.851123 12247 raft_consensus.cc:2379] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 1: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
W20260808 20:13:26.853386 12101 leader_election.cc:341] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:26.854607 12043 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:26.855494 12309 tablet_bootstrap.cc:654] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:26.856590 12000 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:13:26.857915 12469 leader_election.cc:288] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:26.858130 12472 raft_consensus.cc:2781] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:26.858825 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:13:26.859982 12205 leader_election.cc:341] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:26.858955 12472 raft_consensus.cc:686] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:26.861117 12472 consensus_queue.cc:237] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.861621 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:26.862833 12102 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:26.864269 12578 raft_consensus.cc:2781] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:26.864943 12578 raft_consensus.cc:686] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:26.869210 12101 leader_election.cc:302] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:26.871117 12579 raft_consensus.cc:3037] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:26.872691 12576 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:13:26.872706 11994 leader_election.cc:341] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:26.869620 12578 consensus_queue.cc:237] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.874181 12311 tablet_bootstrap.cc:492] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:26.874212 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:26.874577 12311 ts_tablet_manager.cc:1397] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.250s user 0.006s sys 0.008s
I20260808 20:13:26.874827 12251 raft_consensus.cc:3037] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:26.877106 12569 consensus_queue.cc:1035] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:26.878914 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:26.883312 12043 raft_consensus.cc:3037] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
W20260808 20:13:26.886845 11966 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 20:13:26.884518 12251 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:13:26.891721 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:26.896468 12311 raft_consensus.cc:348] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.897773 12311 raft_consensus.cc:374] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:26.898044 12311 raft_consensus.cc:729] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:26.898627 12311 consensus_queue.cc:260] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.896222 12101 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:26.900862 12578 raft_consensus.cc:2781] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:26.901345 12578 raft_consensus.cc:686] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:26.902138 12578 consensus_queue.cc:237] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.904532 12579 raft_consensus.cc:2726] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:26.904745 12359 raft_consensus.cc:482] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:26.905378 12359 raft_consensus.cc:504] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.908995 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:26.909657 12045 raft_consensus.cc:2454] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:26.911203 12043 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:13:26.907481 12359 leader_election.cc:288] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:26.917253 12102 leader_election.cc:302] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:13:26.919114 12311 ts_tablet_manager.cc:1428] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.044s user 0.005s sys 0.001s
I20260808 20:13:26.924573 12311 tablet_bootstrap.cc:492] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:26.925215 12579 raft_consensus.cc:2781] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:26.933442 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:26.934058 12249 raft_consensus.cc:2454] T a383bcab83014009b5698c63cdac298b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:26.929736 12579 raft_consensus.cc:482] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:26.938361 12579 raft_consensus.cc:3037] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:26.940745 12469 consensus_queue.cc:1035] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:26.944224 12313 tablet_bootstrap.cc:492] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:26.944679 12313 ts_tablet_manager.cc:1397] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.332s user 0.022s sys 0.033s
I20260808 20:13:26.959262 12313 raft_consensus.cc:348] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.960690 12313 raft_consensus.cc:374] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:26.961066 12313 raft_consensus.cc:729] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:26.962302 12313 consensus_queue.cc:260] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.957432 12579 raft_consensus.cc:504] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.971092 12313 ts_tablet_manager.cc:1428] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20260808 20:13:26.972285 12313 tablet_bootstrap.cc:492] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:26.975186 12150 raft_consensus.cc:2454] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:13:26.988579 12311 tablet_bootstrap.cc:654] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:26.990090 12309 tablet_bootstrap.cc:492] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:26.990513 12309 ts_tablet_manager.cc:1397] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.158s user 0.007s sys 0.007s
I20260808 20:13:26.990419 12206 leader_election.cc:302] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:26.992147 12321 raft_consensus.cc:2781] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:26.992622 12321 raft_consensus.cc:686] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:26.993386 12321 consensus_queue.cc:237] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:26.993693 12470 consensus_queue.cc:1035] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:26.995579 12579 leader_election.cc:288] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:27.000262 12313 tablet_bootstrap.cc:654] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.001294 12309 raft_consensus.cc:348] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.001506 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:27.002131 12250 raft_consensus.cc:3037] T a383bcab83014009b5698c63cdac298b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.002873 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:27.003515 12043 raft_consensus.cc:3037] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.008455 12250 raft_consensus.cc:2454] T a383bcab83014009b5698c63cdac298b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:27.009549 12101 leader_election.cc:302] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:27.010299 12579 raft_consensus.cc:2781] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:27.010787 12579 raft_consensus.cc:686] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:27.010974 12043 raft_consensus.cc:2454] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:27.011550 12579 consensus_queue.cc:237] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.014134 12309 raft_consensus.cc:374] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.014417 12309 raft_consensus.cc:729] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.016067 12309 consensus_queue.cc:260] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.019079 12311 tablet_bootstrap.cc:492] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:27.019567 12311 ts_tablet_manager.cc:1397] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.095s user 0.007s sys 0.002s
I20260808 20:13:27.020625 12309 ts_tablet_manager.cc:1428] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.030s user 0.005s sys 0.001s
I20260808 20:13:27.022758 12309 tablet_bootstrap.cc:492] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:27.027599 11896 catalog_manager.cc:5495] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 0 to 1, leader changed from <none> to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:13:27.040688 12313 tablet_bootstrap.cc:492] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:27.041164 12313 ts_tablet_manager.cc:1397] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.069s user 0.012s sys 0.018s
I20260808 20:13:27.042037 12311 raft_consensus.cc:348] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.042799 12311 raft_consensus.cc:374] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.043066 12311 raft_consensus.cc:729] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.043787 12309 tablet_bootstrap.cc:654] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.043622 12311 consensus_queue.cc:260] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.045416 12311 ts_tablet_manager.cc:1428] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.026s user 0.008s sys 0.002s
I20260808 20:13:27.046272 12311 tablet_bootstrap.cc:492] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:27.050920 12311 tablet_bootstrap.cc:654] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.054952 12313 raft_consensus.cc:348] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.055620 12313 raft_consensus.cc:374] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.055856 12313 raft_consensus.cc:729] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.055781 12483 consensus_queue.cc:1035] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:27.056643 12313 consensus_queue.cc:260] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.058956 12313 ts_tablet_manager.cc:1428] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260808 20:13:27.060549 12313 tablet_bootstrap.cc:492] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:27.065485 12579 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:27.065938 12579 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.067809 12579 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:27.068866 12313 tablet_bootstrap.cc:654] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.070137 12311 tablet_bootstrap.cc:492] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:27.070571 12311 ts_tablet_manager.cc:1397] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.002s
I20260808 20:13:27.071472 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:27.071821 12472 consensus_queue.cc:1035] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:27.073266 12311 raft_consensus.cc:348] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.073908 12311 raft_consensus.cc:374] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.074177 12311 raft_consensus.cc:729] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.074427 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:27.074708 12311 consensus_queue.cc:260] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.075098 12251 raft_consensus.cc:2454] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:27.077260 12101 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:27.079244 12311 ts_tablet_manager.cc:1428] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260808 20:13:27.080235 12311 tablet_bootstrap.cc:492] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:27.079276 11894 catalog_manager.cc:5495] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
W20260808 20:13:27.082237 12102 leader_election.cc:341] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:27.082252 12576 raft_consensus.cc:2781] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:27.083724 12576 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:27.084374 12576 raft_consensus.cc:3037] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.091542 12311 tablet_bootstrap.cc:654] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.091789 11894 catalog_manager.cc:5495] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:27.094729 12472 raft_consensus.cc:482] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:27.095186 12472 raft_consensus.cc:504] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.097252 12472 leader_election.cc:288] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:27.099037 12576 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.099457 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:27.099686 12309 tablet_bootstrap.cc:492] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:27.100219 12309 ts_tablet_manager.cc:1397] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.078s user 0.007s sys 0.019s
I20260808 20:13:27.101836 12313 tablet_bootstrap.cc:492] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:27.102288 12313 ts_tablet_manager.cc:1397] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.042s user 0.014s sys 0.013s
I20260808 20:13:27.103040 12150 raft_consensus.cc:2379] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 1: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:13:27.102659 12309 raft_consensus.cc:348] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.103394 12309 raft_consensus.cc:374] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.103677 12309 raft_consensus.cc:729] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.105340 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:27.105715 12309 consensus_queue.cc:260] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.104516 12313 raft_consensus.cc:348] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.106959 12313 raft_consensus.cc:374] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.107223 12313 raft_consensus.cc:729] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.108248 12313 consensus_queue.cc:260] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.109287 12309 ts_tablet_manager.cc:1428] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.009s user 0.005s sys 0.002s
I20260808 20:13:27.110144 12309 tablet_bootstrap.cc:492] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:27.110843 12313 ts_tablet_manager.cc:1428] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260808 20:13:27.111828 12313 tablet_bootstrap.cc:492] T 003bc27b07fa40e48c1e57a13b578981 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:27.108523 12000 leader_election.cc:302] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:27.115880 11897 catalog_manager.cc:5495] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:27.116591 12472 raft_consensus.cc:3037] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.117887 11897 catalog_manager.cc:5495] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 2, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 2 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:27.120452 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:13:27.121801 12102 leader_election.cc:341] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:27.123349 12472 raft_consensus.cc:2726] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:27.124423 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:27.125056 12250 raft_consensus.cc:3037] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.127275 12313 tablet_bootstrap.cc:654] T 003bc27b07fa40e48c1e57a13b578981 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.129166 12576 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:27.130003 12309 tablet_bootstrap.cc:654] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.131862 12250 raft_consensus.cc:2454] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:27.132997 12101 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:27.134035 12576 raft_consensus.cc:2781] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:27.134632 12576 raft_consensus.cc:686] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:27.135800 12311 tablet_bootstrap.cc:492] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:27.135408 12576 consensus_queue.cc:237] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.136358 12311 ts_tablet_manager.cc:1397] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.056s user 0.016s sys 0.010s
I20260808 20:13:27.138944 12311 raft_consensus.cc:348] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.139659 12311 raft_consensus.cc:374] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.139945 12311 raft_consensus.cc:729] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.140625 12311 consensus_queue.cc:260] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.143119 12311 ts_tablet_manager.cc:1428] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20260808 20:13:27.144018 12311 tablet_bootstrap.cc:492] T c4dc15f09b4a4470ac96054b70602187 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:27.146395 12309 tablet_bootstrap.cc:492] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:27.146871 12309 ts_tablet_manager.cc:1397] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.006s
I20260808 20:13:27.150704 12311 tablet_bootstrap.cc:654] T c4dc15f09b4a4470ac96054b70602187 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.149542 12309 raft_consensus.cc:348] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.152097 12309 raft_consensus.cc:374] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.152343 12309 raft_consensus.cc:729] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.153625 11896 catalog_manager.cc:5495] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 0 to 1, leader changed from <none> to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:13:27.179383 11897 catalog_manager.cc:5495] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:27.183676 11897 catalog_manager.cc:5495] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:27.195608 12321 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:27.196139 12321 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.205078 12321 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:27.214512 12309 consensus_queue.cc:260] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.215206 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:27.218564 12309 ts_tablet_manager.cc:1428] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.071s user 0.020s sys 0.040s
I20260808 20:13:27.219462 12309 tablet_bootstrap.cc:492] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:27.220568 12313 tablet_bootstrap.cc:492] T 003bc27b07fa40e48c1e57a13b578981 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:27.221038 12313 ts_tablet_manager.cc:1397] T 003bc27b07fa40e48c1e57a13b578981 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.109s user 0.011s sys 0.006s
I20260808 20:13:27.223734 12313 raft_consensus.cc:348] T 003bc27b07fa40e48c1e57a13b578981 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.224545 12313 raft_consensus.cc:374] T 003bc27b07fa40e48c1e57a13b578981 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.224857 12313 raft_consensus.cc:729] T 003bc27b07fa40e48c1e57a13b578981 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.224876 12309 tablet_bootstrap.cc:654] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.225718 12313 consensus_queue.cc:260] T 003bc27b07fa40e48c1e57a13b578981 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.229485 12576 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:27.230051 12576 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.234472 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:27.236773 12313 ts_tablet_manager.cc:1428] T 003bc27b07fa40e48c1e57a13b578981 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.015s user 0.002s sys 0.004s
W20260808 20:13:27.237650 12102 leader_election.cc:341] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:27.238137 12313 tablet_bootstrap.cc:492] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:27.247401 12576 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:27.249614 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:13:27.251094 12101 leader_election.cc:341] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:27.251556 12101 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:27.253353 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:27.254210 12045 raft_consensus.cc:2379] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 1: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:13:27.255383 12205 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:27.256201 12313 tablet_bootstrap.cc:654] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.256170 12321 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.256707 12576 raft_consensus.cc:2726] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:27.261662 12313 tablet_bootstrap.cc:492] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:27.261660 12311 tablet_bootstrap.cc:492] T c4dc15f09b4a4470ac96054b70602187 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:27.262234 12313 ts_tablet_manager.cc:1397] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.002s
I20260808 20:13:27.262290 12311 ts_tablet_manager.cc:1397] T c4dc15f09b4a4470ac96054b70602187 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.118s user 0.056s sys 0.046s
I20260808 20:13:27.262529 12321 raft_consensus.cc:2726] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:27.262900 12309 tablet_bootstrap.cc:492] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:27.263303 12309 ts_tablet_manager.cc:1397] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.044s user 0.016s sys 0.018s
I20260808 20:13:27.264593 12313 raft_consensus.cc:348] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.264744 12311 raft_consensus.cc:348] T c4dc15f09b4a4470ac96054b70602187 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.264926 12309 raft_consensus.cc:348] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.265281 12313 raft_consensus.cc:374] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.265409 12311 raft_consensus.cc:374] T c4dc15f09b4a4470ac96054b70602187 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.265523 12309 raft_consensus.cc:374] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.265580 12313 raft_consensus.cc:729] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.265695 12311 raft_consensus.cc:729] T c4dc15f09b4a4470ac96054b70602187 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.265869 12309 raft_consensus.cc:729] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.266323 12313 consensus_queue.cc:260] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.266394 12311 consensus_queue.cc:260] T c4dc15f09b4a4470ac96054b70602187 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.266552 12309 consensus_queue.cc:260] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.268409 12311 ts_tablet_manager.cc:1428] T c4dc15f09b4a4470ac96054b70602187 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20260808 20:13:27.268520 12309 ts_tablet_manager.cc:1428] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260808 20:13:27.269740 12309 tablet_bootstrap.cc:492] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:27.275135 12309 tablet_bootstrap.cc:654] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.280781 12311 tablet_bootstrap.cc:492] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:27.285147 12313 ts_tablet_manager.cc:1428] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260808 20:13:27.285956 12313 tablet_bootstrap.cc:492] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:27.291818 12313 tablet_bootstrap.cc:654] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.293485 12311 tablet_bootstrap.cc:654] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.302841 12309 tablet_bootstrap.cc:492] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:27.303236 12309 ts_tablet_manager.cc:1397] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.034s user 0.015s sys 0.015s
I20260808 20:13:27.304399 12472 consensus_queue.cc:1035] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:27.305475 12309 raft_consensus.cc:348] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.306126 12309 raft_consensus.cc:374] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.306353 12309 raft_consensus.cc:729] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.307250 12045 raft_consensus.cc:1261] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260808 20:13:27.306972 12309 consensus_queue.cc:260] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.308789 12576 consensus_queue.cc:1035] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:27.314949 12309 ts_tablet_manager.cc:1428] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260808 20:13:27.315778 12309 tablet_bootstrap.cc:492] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:27.321240 12309 tablet_bootstrap.cc:654] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.327476 12250 raft_consensus.cc:1261] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260808 20:13:27.330489 12579 consensus_queue.cc:1035] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:27.331779 12579 consensus_queue.cc:1035] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:27.341204 12321 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:27.341689 12321 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.344635 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:13:27.347177 12205 leader_election.cc:341] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:27.355662 12472 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:27.356197 12472 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.360702 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:13:27.362248 11994 leader_election.cc:341] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:27.362874 12472 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:27.370009 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:27.370707 12152 raft_consensus.cc:2454] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 0.
I20260808 20:13:27.371991 12206 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:27.372920 12470 raft_consensus.cc:2781] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:27.373286 12470 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:27.373656 12470 raft_consensus.cc:3037] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.362633 12579 consensus_queue.cc:1035] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:27.368700 12321 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:27.385741 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:27.386541 12150 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:27.387872 12000 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:27.389150 12386 raft_consensus.cc:2781] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:27.389592 12386 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:27.389950 12386 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.391232 12321 consensus_queue.cc:1035] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:27.396126 12321 consensus_queue.cc:1035] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:27.401518 12470 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.403549 12386 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.404417 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:27.415958 12251 raft_consensus.cc:3037] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.422370 12470 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:13:27.423218 12205 leader_election.cc:341] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:27.424299 12472 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:27.424778 12472 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.426676 12472 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:27.428721 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:27.429538 12152 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:27.430626 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:27.431255 12249 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:27.434857 12309 tablet_bootstrap.cc:492] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:27.435287 12309 ts_tablet_manager.cc:1397] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.120s user 0.014s sys 0.067s
I20260808 20:13:27.435238 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:27.441375 12146 raft_consensus.cc:3037] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.439857 12000 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:13:27.451229 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:27.451768 12150 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.453065 12475 raft_consensus.cc:2781] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:27.453436 12475 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:27.453759 12475 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.457819 12483 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:27.458241 12483 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.460072 12483 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:27.461124 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:27.461079 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:27.462050 12249 raft_consensus.cc:2379] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 1: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:13:27.463112 11994 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:27.463904 12483 raft_consensus.cc:3037] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.467763 12150 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:27.474911 12386 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:27.486333 12483 raft_consensus.cc:2726] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:27.486594 12321 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:27.486517 12311 tablet_bootstrap.cc:492] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:27.487092 12321 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.487663 12311 ts_tablet_manager.cc:1397] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.207s user 0.016s sys 0.008s
I20260808 20:13:27.492286 12475 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.502581 12475 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:27.495029 12311 raft_consensus.cc:348] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.496142 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:27.496555 12321 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:27.499643 12469 consensus_queue.cc:1035] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:27.500450 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:27.500588 12000 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:13:27.508170 12205 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:27.508315 12475 raft_consensus.cc:2781] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:27.509034 12321 raft_consensus.cc:2726] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:13:27.503468 12311 raft_consensus.cc:374] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.478492 12146 raft_consensus.cc:2454] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:13:27.512276 12311 raft_consensus.cc:729] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.512073 12475 raft_consensus.cc:686] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:27.515545 12313 tablet_bootstrap.cc:492] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:27.515640 12475 consensus_queue.cc:237] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.517925 12313 ts_tablet_manager.cc:1397] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.232s user 0.073s sys 0.081s
I20260808 20:13:27.516285 12311 consensus_queue.cc:260] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.524919 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:27.528188 12152 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.530007 12309 raft_consensus.cc:348] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.530609 12309 raft_consensus.cc:374] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.530768 12309 raft_consensus.cc:729] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.531231 12309 consensus_queue.cc:260] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.532056 12311 ts_tablet_manager.cc:1428] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.044s user 0.006s sys 0.001s
I20260808 20:13:27.532928 12311 tablet_bootstrap.cc:492] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:27.533432 12309 ts_tablet_manager.cc:1428] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.098s user 0.005s sys 0.000s
I20260808 20:13:27.534240 12309 tablet_bootstrap.cc:492] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:27.515528 12206 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:27.537284 12573 consensus_queue.cc:1035] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260808 20:13:27.541124 12321 raft_consensus.cc:2781] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:27.543459 12321 raft_consensus.cc:686] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:27.544618 12321 consensus_queue.cc:237] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.546236 12313 raft_consensus.cc:348] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.547015 12313 raft_consensus.cc:374] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.547322 12313 raft_consensus.cc:729] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.547966 12313 consensus_queue.cc:260] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.549503 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:27.550782 12573 raft_consensus.cc:482] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:27.558537 12313 ts_tablet_manager.cc:1428] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.035s user 0.002s sys 0.003s
I20260808 20:13:27.553936 12569 consensus_queue.cc:1035] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:27.558239 12573 raft_consensus.cc:504] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.559919 12389 consensus_queue.cc:1035] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:27.551842 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:27.560999 12249 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.561833 12313 tablet_bootstrap.cc:492] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:27.562879 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:27.563500 12047 raft_consensus.cc:2454] T c4dc15f09b4a4470ac96054b70602187 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:27.564563 12102 leader_election.cc:302] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:13:27.561872 12579 consensus_queue.cc:1035] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260808 20:13:27.568074 12488 raft_consensus.cc:2781] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:27.568092 12389 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:27.560353 12573 leader_election.cc:288] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:27.570240 12389 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.572290 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:27.568535 12488 raft_consensus.cc:482] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:27.573427 12389 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:27.573632 12488 raft_consensus.cc:3037] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.574872 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:27.565416 12152 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
W20260808 20:13:27.579981 12101 leader_election.cc:341] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:27.580164 12573 consensus_queue.cc:1035] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 20:13:27.582100 11994 leader_election.cc:341] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
W20260808 20:13:27.583568 12205 leader_election.cc:341] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:27.586920 12000 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:13:27.587714 12475 raft_consensus.cc:2781] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:27.590550 12475 raft_consensus.cc:686] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:27.587978 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:27.592011 12475 consensus_queue.cc:237] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.595608 12311 tablet_bootstrap.cc:654] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.595427 11896 catalog_manager.cc:5495] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 0 to 1, leader changed from <none> to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
W20260808 20:13:27.598133 12206 leader_election.cc:341] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:27.598613 12206 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:27.601151 12321 raft_consensus.cc:2726] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:27.602736 12488 raft_consensus.cc:504] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.604616 12488 leader_election.cc:288] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:27.613065 11894 catalog_manager.cc:5495] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:27.605650 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:27.619199 12047 raft_consensus.cc:3037] T c4dc15f09b4a4470ac96054b70602187 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.622869 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:27.621811 12313 tablet_bootstrap.cc:654] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.627076 12249 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
W20260808 20:13:27.632460 12101 leader_election.cc:341] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:27.658677 12309 tablet_bootstrap.cc:654] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.670148 12311 tablet_bootstrap.cc:492] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:27.670667 12311 ts_tablet_manager.cc:1397] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.138s user 0.012s sys 0.002s
I20260808 20:13:27.683276 12047 raft_consensus.cc:2454] T c4dc15f09b4a4470ac96054b70602187 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:27.685745 12488 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:27.686209 12488 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.688048 12488 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:27.695252 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:27.696444 12043 raft_consensus.cc:2454] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:27.697570 12102 leader_election.cc:302] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:27.698622 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:27.698901 12578 raft_consensus.cc:2781] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:27.699393 12578 raft_consensus.cc:686] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:27.700148 12578 consensus_queue.cc:237] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.701151 12247 raft_consensus.cc:2454] T 003bc27b07fa40e48c1e57a13b578981 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:27.702500 12101 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:27.703228 12488 raft_consensus.cc:2781] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:27.703565 12488 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:27.703879 12488 raft_consensus.cc:3037] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.706375 12311 raft_consensus.cc:348] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.709024 12311 raft_consensus.cc:374] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.709667 12311 raft_consensus.cc:729] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.710951 12311 consensus_queue.cc:260] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.716958 12386 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:27.717329 12311 ts_tablet_manager.cc:1428] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.046s user 0.005s sys 0.001s
I20260808 20:13:27.717391 12386 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.717928 12313 tablet_bootstrap.cc:492] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:27.718425 12313 ts_tablet_manager.cc:1397] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.157s user 0.013s sys 0.010s
I20260808 20:13:27.718415 12311 tablet_bootstrap.cc:492] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:27.719888 12386 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:27.720606 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:27.715776 11896 catalog_manager.cc:5495] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:27.720912 12313 raft_consensus.cc:348] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.722299 12313 raft_consensus.cc:374] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.722572 12313 raft_consensus.cc:729] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.722115 12488 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.722944 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:27.723219 12313 consensus_queue.cc:260] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.724393 12309 tablet_bootstrap.cc:492] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:27.724933 12309 ts_tablet_manager.cc:1397] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.191s user 0.009s sys 0.007s
I20260808 20:13:27.729031 12313 ts_tablet_manager.cc:1428] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.010s user 0.005s sys 0.002s
I20260808 20:13:27.729889 12313 tablet_bootstrap.cc:492] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:27.731524 12000 leader_election.cc:302] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:27.732620 12472 raft_consensus.cc:2726] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:27.735010 12313 tablet_bootstrap.cc:654] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.740871 12309 raft_consensus.cc:348] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.741830 12309 raft_consensus.cc:374] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.742203 12309 raft_consensus.cc:729] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.740872 12578 consensus_queue.cc:1035] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260808 20:13:27.745064 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:27.745657 12247 raft_consensus.cc:3037] T 003bc27b07fa40e48c1e57a13b578981 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.746220 12309 consensus_queue.cc:260] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.749101 12309 ts_tablet_manager.cc:1428] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.024s user 0.003s sys 0.003s
I20260808 20:13:27.750554 12311 tablet_bootstrap.cc:654] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.752306 12247 raft_consensus.cc:2454] T 003bc27b07fa40e48c1e57a13b578981 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:27.751397 12309 tablet_bootstrap.cc:492] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:27.753950 12101 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:27.753473 11897 catalog_manager.cc:5495] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:27.754766 12578 raft_consensus.cc:2781] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:27.755378 12578 raft_consensus.cc:686] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:27.756909 12488 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:27.756162 12578 consensus_queue.cc:237] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.771490 12313 tablet_bootstrap.cc:492] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:27.774443 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:27.775123 12045 raft_consensus.cc:3037] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.781172 12046 raft_consensus.cc:3037] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.788542 12313 ts_tablet_manager.cc:1397] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.059s user 0.015s sys 0.008s
I20260808 20:13:27.791815 12313 raft_consensus.cc:348] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.792450 12313 raft_consensus.cc:374] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.792747 12313 raft_consensus.cc:729] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.793195 12045 raft_consensus.cc:2454] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:27.793357 12313 consensus_queue.cc:260] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.795159 12313 ts_tablet_manager.cc:1428] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260808 20:13:27.795926 12313 tablet_bootstrap.cc:492] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:27.807089 12311 tablet_bootstrap.cc:492] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:27.807518 12311 ts_tablet_manager.cc:1397] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.089s user 0.015s sys 0.021s
I20260808 20:13:27.807526 12313 tablet_bootstrap.cc:654] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.810144 12311 raft_consensus.cc:348] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.810854 12311 raft_consensus.cc:374] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.811095 12311 raft_consensus.cc:729] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.811655 11897 catalog_manager.cc:5495] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:27.811661 12311 consensus_queue.cc:260] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.815971 12311 ts_tablet_manager.cc:1428] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260808 20:13:27.816679 12309 tablet_bootstrap.cc:654] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.816799 12311 tablet_bootstrap.cc:492] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:27.831091 12311 tablet_bootstrap.cc:654] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.833791 12488 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:27.834306 12488 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.834868 12309 tablet_bootstrap.cc:492] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:27.835302 12309 ts_tablet_manager.cc:1397] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.084s user 0.010s sys 0.006s
I20260808 20:13:27.836225 12488 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:27.837617 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:27.838222 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:13:27.839211 12101 leader_election.cc:341] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
W20260808 20:13:27.840569 12102 leader_election.cc:341] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:27.841075 12313 tablet_bootstrap.cc:492] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:27.840986 12102 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:27.841665 12488 raft_consensus.cc:2726] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:27.837528 12309 raft_consensus.cc:348] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.842067 12313 ts_tablet_manager.cc:1397] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.046s user 0.013s sys 0.003s
I20260808 20:13:27.842159 12309 raft_consensus.cc:374] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.842501 12309 raft_consensus.cc:729] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.842998 12309 consensus_queue.cc:260] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.844499 12309 ts_tablet_manager.cc:1428] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260808 20:13:27.844305 12313 raft_consensus.cc:348] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.844980 12313 raft_consensus.cc:374] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.845238 12313 raft_consensus.cc:729] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.845892 12309 tablet_bootstrap.cc:492] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:27.845809 12313 consensus_queue.cc:260] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.847863 12313 ts_tablet_manager.cc:1428] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260808 20:13:27.849948 12311 tablet_bootstrap.cc:492] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:27.850404 12311 ts_tablet_manager.cc:1397] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.003s
I20260808 20:13:27.852656 12313 tablet_bootstrap.cc:492] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:27.853140 12311 raft_consensus.cc:348] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.853789 12311 raft_consensus.cc:374] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.854059 12311 raft_consensus.cc:729] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.854678 12311 consensus_queue.cc:260] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.856707 12311 ts_tablet_manager.cc:1428] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 20:13:27.857963 12313 tablet_bootstrap.cc:654] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.858212 12311 tablet_bootstrap.cc:492] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:27.864133 12311 tablet_bootstrap.cc:654] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.872706 12309 tablet_bootstrap.cc:654] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.875247 12311 tablet_bootstrap.cc:492] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:27.875703 12311 ts_tablet_manager.cc:1397] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.013s
I20260808 20:13:27.876149 12313 tablet_bootstrap.cc:492] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:27.876509 12313 ts_tablet_manager.cc:1397] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.006s
I20260808 20:13:27.877918 12311 raft_consensus.cc:348] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.878661 12311 raft_consensus.cc:374] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.878727 12313 raft_consensus.cc:348] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.878963 12311 raft_consensus.cc:729] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.879271 12313 raft_consensus.cc:374] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.879319 12309 tablet_bootstrap.cc:492] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:27.879595 12313 raft_consensus.cc:729] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.879878 12309 ts_tablet_manager.cc:1397] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.004s
I20260808 20:13:27.879702 12311 consensus_queue.cc:260] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.880165 12313 consensus_queue.cc:260] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.881794 12311 ts_tablet_manager.cc:1428] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260808 20:13:27.882033 12313 ts_tablet_manager.cc:1428] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260808 20:13:27.882998 12311 tablet_bootstrap.cc:492] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:27.884178 12313 tablet_bootstrap.cc:492] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:27.891570 12309 raft_consensus.cc:348] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.892228 12309 raft_consensus.cc:374] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.892473 12309 raft_consensus.cc:729] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.893107 12309 consensus_queue.cc:260] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.893776 12311 tablet_bootstrap.cc:654] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.894179 12313 tablet_bootstrap.cc:654] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.900043 12309 ts_tablet_manager.cc:1428] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.020s user 0.003s sys 0.003s
I20260808 20:13:27.901043 12309 tablet_bootstrap.cc:492] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:27.906948 12309 tablet_bootstrap.cc:654] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.907455 12313 tablet_bootstrap.cc:492] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:27.907936 12313 ts_tablet_manager.cc:1397] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.004s
I20260808 20:13:27.910240 12313 raft_consensus.cc:348] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.911094 12313 raft_consensus.cc:374] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.911412 12313 raft_consensus.cc:729] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.912074 12313 consensus_queue.cc:260] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.916327 12313 ts_tablet_manager.cc:1428] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260808 20:13:27.917219 12313 tablet_bootstrap.cc:492] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:27.917428 12488 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:27.917922 12488 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.919858 12488 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:27.921387 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:27.922122 12250 raft_consensus.cc:2454] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:27.922780 12313 tablet_bootstrap.cc:654] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:27.923309 12578 raft_consensus.cc:482] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:27.923799 12578 raft_consensus.cc:504] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.925806 12101 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:27.926680 12576 raft_consensus.cc:2781] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:27.927048 12576 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:27.927359 12576 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.928788 12578 leader_election.cc:288] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:27.933302 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:27.934015 12047 raft_consensus.cc:2454] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:27.932669 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:27.935639 12046 raft_consensus.cc:2454] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:27.938063 12102 leader_election.cc:302] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:13:27.938963 12578 raft_consensus.cc:2781] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:27.939338 12578 raft_consensus.cc:482] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:27.939649 12578 raft_consensus.cc:3037] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.917387 12311 tablet_bootstrap.cc:492] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:27.943116 12311 ts_tablet_manager.cc:1397] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.060s user 0.019s sys 0.001s
I20260808 20:13:27.944145 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:27.945537 12250 raft_consensus.cc:2454] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:27.946547 12469 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:27.947026 12469 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.949822 12469 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:27.950790 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:13:27.953362 12205 leader_election.cc:341] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:27.954316 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:13:27.957263 12206 leader_election.cc:341] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:27.957664 12206 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:27.958602 12469 raft_consensus.cc:2726] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:27.964949 12472 consensus_queue.cc:1035] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:27.970737 12576 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.973743 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:27.974422 12250 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.977415 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:27.977981 12046 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:27.980810 12250 raft_consensus.cc:2454] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:27.983407 12101 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:27.972767 12576 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:27.984130 12488 raft_consensus.cc:2781] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:27.984577 12488 raft_consensus.cc:686] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:27.985375 12488 consensus_queue.cc:237] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.986330 12573 raft_consensus.cc:482] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:27.986824 12573 raft_consensus.cc:504] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.991653 12311 raft_consensus.cc:348] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:27.992367 12311 raft_consensus.cc:374] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:27.992623 12311 raft_consensus.cc:729] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:27.994486 12573 leader_election.cc:288] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:27.995249 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:13:28.003720 12102 leader_election.cc:341] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:28.005175 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:13:28.006440 12101 leader_election.cc:341] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:28.006834 12101 leader_election.cc:302] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:28.007640 12488 raft_consensus.cc:2726] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:27.993438 12311 consensus_queue.cc:260] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:13:28.018776 11994 consensus_peers.cc:487] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 -> Peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Couldn't send request to peer b9d94f35ea15421aa1d721201c782d8a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:28.019253 12047 raft_consensus.cc:3037] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:28.024979 12309 tablet_bootstrap.cc:492] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:28.025549 12309 ts_tablet_manager.cc:1397] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.125s user 0.033s sys 0.040s
I20260808 20:13:28.026139 12313 tablet_bootstrap.cc:492] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:28.026566 12313 ts_tablet_manager.cc:1397] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.109s user 0.021s sys 0.053s
I20260808 20:13:28.030184 12664 consensus_queue.cc:1035] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:28.033808 12046 raft_consensus.cc:2454] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:28.034139 12309 raft_consensus.cc:348] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.034901 12309 raft_consensus.cc:374] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:28.035205 12309 raft_consensus.cc:729] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:28.034400 11896 catalog_manager.cc:5495] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:28.037534 12311 ts_tablet_manager.cc:1428] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.094s user 0.006s sys 0.000s
I20260808 20:13:28.036818 12321 consensus_queue.cc:1035] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:28.014107 12578 raft_consensus.cc:504] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.035827 12309 consensus_queue.cc:260] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.040409 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:28.041033 12251 raft_consensus.cc:3037] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:28.041324 12313 raft_consensus.cc:348] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.041932 12313 raft_consensus.cc:374] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:28.042196 12313 raft_consensus.cc:729] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:28.042970 12309 ts_tablet_manager.cc:1428] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.017s user 0.004s sys 0.002s
I20260808 20:13:28.043800 12309 tablet_bootstrap.cc:492] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:28.046222 12313 consensus_queue.cc:260] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.048274 12313 ts_tablet_manager.cc:1428] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20260808 20:13:28.048507 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:28.049134 12309 tablet_bootstrap.cc:654] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:28.049093 12045 raft_consensus.cc:3037] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:28.051433 12578 leader_election.cc:288] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:28.055902 12251 raft_consensus.cc:2454] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:28.059556 12101 leader_election.cc:302] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:28.057901 12045 raft_consensus.cc:2454] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:28.051147 12311 tablet_bootstrap.cc:492] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:28.063356 12313 tablet_bootstrap.cc:492] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:28.067723 12469 consensus_queue.cc:1035] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:28.068692 12313 tablet_bootstrap.cc:654] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:28.072021 12573 raft_consensus.cc:2781] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:28.082782 12573 raft_consensus.cc:686] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:28.083567 12573 consensus_queue.cc:237] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.097359 12311 tablet_bootstrap.cc:654] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
W20260808 20:13:28.099062 11966 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 20:13:28.103932 12313 tablet_bootstrap.cc:492] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:28.106405 12313 ts_tablet_manager.cc:1397] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.058s user 0.010s sys 0.000s
I20260808 20:13:28.108713 12313 raft_consensus.cc:348] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.104377 12309 tablet_bootstrap.cc:492] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:28.109748 12309 ts_tablet_manager.cc:1397] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.066s user 0.006s sys 0.030s
I20260808 20:13:28.112286 12311 tablet_bootstrap.cc:492] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:28.112010 12309 raft_consensus.cc:348] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.112697 12311 ts_tablet_manager.cc:1397] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.000s
I20260808 20:13:28.112749 12309 raft_consensus.cc:374] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:28.113265 12309 raft_consensus.cc:729] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:28.113981 12309 consensus_queue.cc:260] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.114230 11894 catalog_manager.cc:5495] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:28.114995 12311 raft_consensus.cc:348] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.115659 12311 raft_consensus.cc:374] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:28.115907 12311 raft_consensus.cc:729] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:28.116528 12309 ts_tablet_manager.cc:1428] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260808 20:13:28.116847 12313 raft_consensus.cc:374] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:28.117201 12313 raft_consensus.cc:729] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:28.116465 12311 consensus_queue.cc:260] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.117833 12309 tablet_bootstrap.cc:492] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:28.118060 12472 consensus_queue.cc:1035] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:28.119467 12311 ts_tablet_manager.cc:1428] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260808 20:13:28.120263 12311 tablet_bootstrap.cc:492] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:28.117792 12313 consensus_queue.cc:260] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.122676 12573 raft_consensus.cc:482] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:28.123098 12573 raft_consensus.cc:504] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.124063 12309 tablet_bootstrap.cc:654] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:28.124979 12573 leader_election.cc:288] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:28.126610 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:28.127216 12250 raft_consensus.cc:2454] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:28.128391 12101 leader_election.cc:302] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:28.130043 12578 raft_consensus.cc:2781] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:28.130383 12578 raft_consensus.cc:482] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:28.130475 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:28.130678 12313 ts_tablet_manager.cc:1428] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20260808 20:13:28.131709 12313 tablet_bootstrap.cc:492] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:28.132685 12573 consensus_queue.cc:1035] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 20:13:28.135277 12102 leader_election.cc:341] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:28.137032 12313 tablet_bootstrap.cc:654] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:28.130731 12578 raft_consensus.cc:3037] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:28.145383 12578 raft_consensus.cc:504] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.147977 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:28.149521 12311 tablet_bootstrap.cc:654] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
W20260808 20:13:28.150291 12102 leader_election.cc:341] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:28.174616 12311 tablet_bootstrap.cc:492] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:28.175110 12311 ts_tablet_manager.cc:1397] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.055s user 0.012s sys 0.000s
I20260808 20:13:28.182572 12251 raft_consensus.cc:3037] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:28.188181 12578 leader_election.cc:288] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:28.194396 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:28.195030 12250 raft_consensus.cc:3037] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:28.198443 12313 tablet_bootstrap.cc:492] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:28.198872 12313 ts_tablet_manager.cc:1397] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.067s user 0.008s sys 0.010s
I20260808 20:13:28.197024 12578 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:28.199406 12578 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.204588 12313 raft_consensus.cc:348] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.205494 12313 raft_consensus.cc:374] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:28.205319 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:28.205765 12313 raft_consensus.cc:729] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:28.205787 12576 consensus_queue.cc:1035] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:28.206561 12313 consensus_queue.cc:260] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.207521 12309 tablet_bootstrap.cc:492] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:28.209019 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:28.208089 12313 ts_tablet_manager.cc:1428] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20260808 20:13:28.209688 12309 ts_tablet_manager.cc:1397] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.092s user 0.023s sys 0.034s
I20260808 20:13:28.210530 12313 tablet_bootstrap.cc:492] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:28.211615 12247 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:28.211206 12578 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:28.211943 12250 raft_consensus.cc:2454] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:28.213289 12101 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:28.215011 12101 leader_election.cc:302] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:28.215737 12576 raft_consensus.cc:2781] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:28.214530 12578 raft_consensus.cc:2781] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:28.216336 12576 raft_consensus.cc:686] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:28.217724 12576 consensus_queue.cc:237] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.216181 12578 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:28.221424 12578 raft_consensus.cc:3037] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:28.224414 12309 raft_consensus.cc:348] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.225129 12309 raft_consensus.cc:374] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:28.225365 12309 raft_consensus.cc:729] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:28.225966 12309 consensus_queue.cc:260] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.228197 12309 ts_tablet_manager.cc:1428] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.018s user 0.004s sys 0.001s
I20260808 20:13:28.229058 12309 tablet_bootstrap.cc:492] T 926cdce6649c4ca59961cfac78392647 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
W20260808 20:13:28.233532 12102 leader_election.cc:341] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:28.237855 12313 tablet_bootstrap.cc:654] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:28.240825 12578 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.241012 12311 raft_consensus.cc:348] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.245134 12311 raft_consensus.cc:374] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:28.245399 12311 raft_consensus.cc:729] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:28.247848 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:28.248411 12251 raft_consensus.cc:3037] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:28.250855 12311 consensus_queue.cc:260] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.243522 12578 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:28.253005 12311 ts_tablet_manager.cc:1428] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.078s user 0.009s sys 0.000s
I20260808 20:13:28.253710 12309 tablet_bootstrap.cc:654] T 926cdce6649c4ca59961cfac78392647 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:28.254154 12311 tablet_bootstrap.cc:492] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:28.255532 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:28.256409 12045 raft_consensus.cc:3037] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:28.255059 11894 catalog_manager.cc:5495] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:28.257688 12251 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:28.260545 12101 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:28.261938 12578 raft_consensus.cc:2781] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:28.264405 12045 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:28.268848 12311 tablet_bootstrap.cc:654] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:28.270735 12578 raft_consensus.cc:686] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:28.272536 12578 consensus_queue.cc:237] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.273521 12313 tablet_bootstrap.cc:492] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:28.274173 12313 ts_tablet_manager.cc:1397] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.064s user 0.014s sys 0.003s
I20260808 20:13:28.271559 12576 consensus_queue.cc:1035] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:28.277168 12313 raft_consensus.cc:348] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.277808 12313 raft_consensus.cc:374] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:28.278059 12313 raft_consensus.cc:729] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:28.278661 12313 consensus_queue.cc:260] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.280642 12313 ts_tablet_manager.cc:1428] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260808 20:13:28.282723 12313 tablet_bootstrap.cc:492] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:28.288384 12313 tablet_bootstrap.cc:654] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:28.291770 11896 catalog_manager.cc:5495] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:28.292536 12309 tablet_bootstrap.cc:492] T 926cdce6649c4ca59961cfac78392647 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:28.296284 12309 ts_tablet_manager.cc:1397] T 926cdce6649c4ca59961cfac78392647 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.067s user 0.009s sys 0.009s
I20260808 20:13:28.297612 12313 tablet_bootstrap.cc:492] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:28.298024 12313 ts_tablet_manager.cc:1397] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.004s
I20260808 20:13:28.299039 12309 raft_consensus.cc:348] T 926cdce6649c4ca59961cfac78392647 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.299716 12309 raft_consensus.cc:374] T 926cdce6649c4ca59961cfac78392647 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:28.299958 12309 raft_consensus.cc:729] T 926cdce6649c4ca59961cfac78392647 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:28.300179 12313 raft_consensus.cc:348] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.300844 12313 raft_consensus.cc:374] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:28.301115 12313 raft_consensus.cc:729] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:28.302389 12313 consensus_queue.cc:260] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.304327 12313 ts_tablet_manager.cc:1428] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260808 20:13:28.306247 12313 tablet_bootstrap.cc:492] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:28.306703 12309 consensus_queue.cc:260] T 926cdce6649c4ca59961cfac78392647 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.307780 12311 tablet_bootstrap.cc:492] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:28.308144 12469 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:28.308630 12469 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.308197 12311 ts_tablet_manager.cc:1397] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.054s user 0.013s sys 0.002s
I20260808 20:13:28.310505 12469 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:28.311403 12313 tablet_bootstrap.cc:654] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:28.313120 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:28.313709 12045 raft_consensus.cc:2454] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 0.
I20260808 20:13:28.314728 12205 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:28.316665 12311 raft_consensus.cc:348] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.317441 12311 raft_consensus.cc:374] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:28.317689 12311 raft_consensus.cc:729] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:28.318107 12469 raft_consensus.cc:2781] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:28.318454 12469 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:28.318749 12469 raft_consensus.cc:3037] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:28.319152 12311 consensus_queue.cc:260] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.321383 12311 ts_tablet_manager.cc:1428] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260808 20:13:28.317682 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:28.322149 12152 raft_consensus.cc:2454] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 0.
I20260808 20:13:28.323647 12311 tablet_bootstrap.cc:492] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:28.324061 12576 consensus_queue.cc:1035] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:28.329859 12309 ts_tablet_manager.cc:1428] T 926cdce6649c4ca59961cfac78392647 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.033s user 0.006s sys 0.001s
I20260808 20:13:28.330416 12311 tablet_bootstrap.cc:654] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:28.330682 12309 tablet_bootstrap.cc:492] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:28.345798 12469 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.348315 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:28.348963 12046 raft_consensus.cc:3037] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:28.349187 12472 raft_consensus.cc:482] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:28.349665 12472 raft_consensus.cc:504] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.352200 12472 leader_election.cc:288] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:28.352443 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:13:28.353718 11994 leader_election.cc:341] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:28.353989 12311 tablet_bootstrap.cc:492] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:28.354453 12311 ts_tablet_manager.cc:1397] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.031s user 0.021s sys 0.000s
I20260808 20:13:28.354948 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:28.355574 12152 raft_consensus.cc:2454] T 926cdce6649c4ca59961cfac78392647 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:28.356773 12311 raft_consensus.cc:348] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.357467 12311 raft_consensus.cc:374] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:28.357714 12311 raft_consensus.cc:729] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:28.358295 12311 consensus_queue.cc:260] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.358559 12000 leader_election.cc:302] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:28.359616 12472 raft_consensus.cc:2781] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:28.359956 12472 raft_consensus.cc:482] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:28.360298 12472 raft_consensus.cc:3037] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:28.360109 12046 raft_consensus.cc:2454] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:13:28.361177 12205 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:28.361873 12389 raft_consensus.cc:2781] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:28.362314 12389 raft_consensus.cc:686] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:28.363061 12389 consensus_queue.cc:237] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.367142 12472 raft_consensus.cc:504] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.368983 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:28.369562 12152 raft_consensus.cc:3037] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:28.369673 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:28.370272 12146 raft_consensus.cc:3037] T 926cdce6649c4ca59961cfac78392647 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:28.373642 12469 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:28.376093 12309 tablet_bootstrap.cc:654] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:28.378809 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:28.379065 12152 raft_consensus.cc:2454] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
W20260808 20:13:28.380671 11994 leader_election.cc:341] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:28.378428 12472 leader_election.cc:288] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:28.382393 12146 raft_consensus.cc:2454] T 926cdce6649c4ca59961cfac78392647 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:28.383414 12000 leader_election.cc:302] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:28.384181 12311 ts_tablet_manager.cc:1428] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20260808 20:13:28.384824 12664 raft_consensus.cc:2781] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:28.385114 12311 tablet_bootstrap.cc:492] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:28.385306 12664 raft_consensus.cc:686] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:28.385797 12469 raft_consensus.cc:482] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:28.386243 12469 raft_consensus.cc:504] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.386133 12664 consensus_queue.cc:237] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.388154 12469 leader_election.cc:288] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:28.390439 12313 tablet_bootstrap.cc:492] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:28.390798 12313 ts_tablet_manager.cc:1397] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.085s user 0.027s sys 0.029s
I20260808 20:13:28.391830 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:13:28.393498 12205 leader_election.cc:341] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:28.393437 12313 raft_consensus.cc:348] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.394174 12313 raft_consensus.cc:374] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:28.394433 12313 raft_consensus.cc:729] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:28.395007 12313 consensus_queue.cc:260] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.395339 12469 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:28.395824 12469 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.397074 12313 ts_tablet_manager.cc:1428] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260808 20:13:28.397610 12469 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:28.397920 12313 tablet_bootstrap.cc:492] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:28.399750 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:28.400410 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:28.400277 11896 catalog_manager.cc:5495] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 0 to 1, leader changed from <none> to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:13:28.402177 12146 raft_consensus.cc:2454] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 0.
I20260808 20:13:28.402485 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:13:28.402485 12205 leader_election.cc:341] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
W20260808 20:13:28.403789 12206 leader_election.cc:341] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:28.404198 12206 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:28.407023 12389 raft_consensus.cc:2726] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:28.407761 12206 leader_election.cc:302] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:28.408365 12311 tablet_bootstrap.cc:654] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:28.408761 12389 raft_consensus.cc:2781] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:28.409142 12389 raft_consensus.cc:482] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:28.409457 12389 raft_consensus.cc:3037] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:28.410225 11894 catalog_manager.cc:5495] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:28.412464 12313 tablet_bootstrap.cc:654] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:28.424787 12389 raft_consensus.cc:504] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.429874 12472 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:28.430518 12472 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.435434 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:28.437181 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:13:28.438931 12205 leader_election.cc:341] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:28.438762 12472 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:28.445384 12311 tablet_bootstrap.cc:492] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:28.445837 12311 ts_tablet_manager.cc:1397] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.005s
I20260808 20:13:28.446372 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
W20260808 20:13:28.439145 12000 leader_election.cc:341] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:28.447453 12146 raft_consensus.cc:3037] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:28.448638 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:13:28.449836 11994 leader_election.cc:341] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:28.450359 11994 leader_election.cc:302] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:28.451713 12472 raft_consensus.cc:2726] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:28.446419 12389 leader_election.cc:288] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:28.454336 12309 tablet_bootstrap.cc:492] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:28.454843 12309 ts_tablet_manager.cc:1397] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.124s user 0.012s sys 0.004s
I20260808 20:13:28.458659 12309 raft_consensus.cc:348] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.458907 12311 raft_consensus.cc:348] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.459455 12309 raft_consensus.cc:374] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:28.459604 12311 raft_consensus.cc:374] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:28.459659 12146 raft_consensus.cc:2454] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:13:28.459787 12309 raft_consensus.cc:729] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:28.459944 12311 raft_consensus.cc:729] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:28.460865 12309 consensus_queue.cc:260] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.461280 12313 tablet_bootstrap.cc:492] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:28.460990 12311 consensus_queue.cc:260] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.461251 12206 leader_election.cc:302] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:28.461907 12313 ts_tablet_manager.cc:1397] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.064s user 0.028s sys 0.010s
I20260808 20:13:28.463311 12309 ts_tablet_manager.cc:1428] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260808 20:13:28.463838 12311 ts_tablet_manager.cc:1428] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260808 20:13:28.464334 12389 raft_consensus.cc:2781] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:28.464876 12389 raft_consensus.cc:686] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:28.466454 12309 tablet_bootstrap.cc:492] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:28.467113 12389 consensus_queue.cc:237] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.465197 12311 tablet_bootstrap.cc:492] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:28.475951 12309 tablet_bootstrap.cc:654] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:28.468772 12313 raft_consensus.cc:348] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.477527 12313 raft_consensus.cc:374] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:28.477771 12313 raft_consensus.cc:729] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:28.478380 12313 consensus_queue.cc:260] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.480613 12313 ts_tablet_manager.cc:1428] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.018s user 0.004s sys 0.000s
I20260808 20:13:28.480558 12472 raft_consensus.cc:482] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:28.481489 12313 tablet_bootstrap.cc:492] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:28.481244 12472 raft_consensus.cc:504] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.486289 12472 leader_election.cc:288] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:28.484969 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:13:28.487967 12000 leader_election.cc:341] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:28.489578 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:13:28.490922 11994 leader_election.cc:341] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:28.492404 12311 tablet_bootstrap.cc:654] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:28.498795 12313 tablet_bootstrap.cc:654] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:28.495234 11994 leader_election.cc:302] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:28.502652 12475 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:28.502830 12389 raft_consensus.cc:482] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:28.503509 12475 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.501432 12472 raft_consensus.cc:2726] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:28.503656 12389 raft_consensus.cc:504] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.505544 12251 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:28.506631 11894 catalog_manager.cc:5495] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 0 to 1, leader changed from <none> to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:13:28.505649 12475 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:28.513145 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:28.513736 12469 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:28.517393 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:28.511597 12321 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:28.522831 12321 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.534199 12321 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:28.530553 12247 raft_consensus.cc:2454] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:28.530862 12389 leader_election.cc:288] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:13:28.540053 12000 leader_election.cc:341] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:28.544811 12573 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:28.545553 12573 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.548621 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:28.555132 12573 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:28.525658 12469 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.556605 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:28.542292 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:13:28.561519 12102 leader_election.cc:341] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:28.564492 12309 tablet_bootstrap.cc:492] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:28.565032 12309 ts_tablet_manager.cc:1397] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.099s user 0.020s sys 0.010s
W20260808 20:13:28.568141 12206 leader_election.cc:341] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:28.568436 11994 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:28.570139 12472 raft_consensus.cc:2781] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:28.570715 12472 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:28.571106 12472 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:28.557717 12469 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:28.572193 12309 raft_consensus.cc:348] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.684206 12309 raft_consensus.cc:374] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:28.684504 12309 raft_consensus.cc:729] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:28.685845 12309 consensus_queue.cc:260] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:13:28.575362 12206 leader_election.cc:341] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
W20260808 20:13:28.690732 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 115452us
I20260808 20:13:28.575026 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:28.587139 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:28.576360 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:28.580967 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:28.691721 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:13:28.687140 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 105160us
I20260808 20:13:28.677950 12472 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.706434 12488 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:28.715282 12488 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:13:28.714586 12206 leader_election.cc:341] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:28.708274 12321 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:13:28.721798 12321 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.698554 12579 consensus_queue.cc:1035] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260808 20:13:28.707782 12250 raft_consensus.cc:2454] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:28.718286 12488 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:28.734617 12321 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:13:28.735185 12205 leader_election.cc:341] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:28.735615 12205 leader_election.cc:302] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:28.736747 12046 raft_consensus.cc:2379] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 1: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:13:28.737902 12321 raft_consensus.cc:2726] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:28.728631 12472 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:28.745093 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:28.736634 12576 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260808 20:13:28.747331 12309 ts_tablet_manager.cc:1428] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.182s user 0.009s sys 0.011s
I20260808 20:13:28.748229 12309 tablet_bootstrap.cc:492] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:28.742910 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:28.753093 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:28.755368 12311 tablet_bootstrap.cc:492] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:28.755800 12311 ts_tablet_manager.cc:1397] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.291s user 0.005s sys 0.021s
W20260808 20:13:28.736673 12205 leader_election.cc:341] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:28.760480 12205 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:28.765302 12313 tablet_bootstrap.cc:492] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:28.765765 12313 ts_tablet_manager.cc:1397] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.284s user 0.013s sys 0.022s
I20260808 20:13:28.768292 12469 raft_consensus.cc:2726] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:28.775977 12321 consensus_queue.cc:1035] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:28.779047 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:28.779907 12244 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
W20260808 20:13:28.782753 12000 leader_election.cc:341] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:28.783015 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:28.771426 12101 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:28.789058 12576 raft_consensus.cc:2781] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:28.789474 12576 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:28.789938 12576 raft_consensus.cc:3037] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:28.790560 12309 tablet_bootstrap.cc:654] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:28.796288 12313 raft_consensus.cc:348] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.796751 12313 raft_consensus.cc:374] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:28.796953 12313 raft_consensus.cc:729] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
W20260808 20:13:28.795437 12101 leader_election.cc:341] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:28.797495 12313 consensus_queue.cc:260] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.808387 12313 ts_tablet_manager.cc:1428] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.042s user 0.007s sys 0.000s
I20260808 20:13:28.809068 12313 tablet_bootstrap.cc:492] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:28.828779 12311 raft_consensus.cc:348] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.829530 12311 raft_consensus.cc:374] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:28.829793 12311 raft_consensus.cc:729] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:28.833527 12311 consensus_queue.cc:260] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.841854 12311 ts_tablet_manager.cc:1428] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.086s user 0.003s sys 0.007s
I20260808 20:13:28.842696 12311 tablet_bootstrap.cc:492] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:28.817669 12244 raft_consensus.cc:2454] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:28.858219 12576 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.877174 12569 raft_consensus.cc:482] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:28.877660 12569 raft_consensus.cc:504] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.870051 12578 consensus_queue.cc:1035] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260808 20:13:28.879873 12569 leader_election.cc:288] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:28.881101 11994 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:28.871785 12205 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:28.888417 12762 raft_consensus.cc:2726] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:13:28.889294 12205 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:28.891219 12469 raft_consensus.cc:2726] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:28.893564 12488 consensus_queue.cc:1035] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:28.895186 12762 consensus_queue.cc:1035] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:28.899333 12576 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:28.901291 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:28.901474 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:28.902362 12247 raft_consensus.cc:3037] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:28.913033 12475 raft_consensus.cc:2781] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:28.914074 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:28.915482 12309 tablet_bootstrap.cc:492] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:28.915953 12309 ts_tablet_manager.cc:1397] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.168s user 0.008s sys 0.020s
I20260808 20:13:28.920398 12475 raft_consensus.cc:686] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:28.925264 12475 consensus_queue.cc:237] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.929353 12313 tablet_bootstrap.cc:654] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:28.934470 12488 raft_consensus.cc:482] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:13:28.929049 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 48196.1us
I20260808 20:13:28.935056 12488 raft_consensus.cc:504] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:28.975919 12488 leader_election.cc:288] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:28.975577 12569 consensus_queue.cc:1035] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:28.976683 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:28.978580 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:29.021517 12311 tablet_bootstrap.cc:654] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
W20260808 20:13:29.023607 12783 log.cc:924] Time spent T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
I20260808 20:13:29.021593 12309 raft_consensus.cc:348] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.026503 12309 raft_consensus.cc:374] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:29.026783 12309 raft_consensus.cc:729] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:29.028244 12046 raft_consensus.cc:3037] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:29.020677 12251 raft_consensus.cc:2454] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:29.030890 12101 leader_election.cc:302] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:29.031998 12488 raft_consensus.cc:2781] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:29.032521 12488 raft_consensus.cc:482] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:29.032877 12488 raft_consensus.cc:3037] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
W20260808 20:13:29.042527 12205 consensus_peers.cc:487] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a -> Peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Couldn't send request to peer 4cd46f13bceb4c3c9ef71e755e17b995. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:28.999401 12247 raft_consensus.cc:2454] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:29.027381 12309 consensus_queue.cc:260] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.044497 12761 consensus_queue.cc:1035] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260808 20:13:29.068635 12309 ts_tablet_manager.cc:1428] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.152s user 0.016s sys 0.020s
I20260808 20:13:29.044519 12101 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:29.073354 12309 tablet_bootstrap.cc:492] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:29.065409 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:29.054123 11994 leader_election.cc:302] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:29.096076 12578 raft_consensus.cc:2781] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:29.112349 12578 raft_consensus.cc:686] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:29.112438 12321 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:29.112978 12321 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.093266 12766 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260808 20:13:29.129173 12321 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:13:29.093881 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 48568.1us
I20260808 20:13:29.150528 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:13:29.146131 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 98517.1us
I20260808 20:13:29.121027 12472 raft_consensus.cc:2726] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:29.159617 12488 raft_consensus.cc:504] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.169576 12488 leader_election.cc:288] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:29.222584 12578 consensus_queue.cc:237] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:13:29.229373 12206 leader_election.cc:341] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
W20260808 20:13:29.241420 12102 leader_election.cc:341] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:29.241907 12576 consensus_queue.cc:1035] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:29.241859 12102 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:29.247437 12313 tablet_bootstrap.cc:492] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:29.247891 12313 ts_tablet_manager.cc:1397] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.439s user 0.019s sys 0.015s
I20260808 20:13:29.251415 12313 raft_consensus.cc:348] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.252290 12313 raft_consensus.cc:374] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:29.252547 12313 raft_consensus.cc:729] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:29.253111 12313 consensus_queue.cc:260] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.241734 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:29.254392 12576 consensus_queue.cc:1035] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:29.255426 12313 ts_tablet_manager.cc:1428] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260808 20:13:29.256270 12313 tablet_bootstrap.cc:492] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:29.248657 12488 raft_consensus.cc:2726] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:29.268229 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:29.269140 12247 raft_consensus.cc:3037] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
W20260808 20:13:29.273800 12102 leader_election.cc:341] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:29.274849 12795 consensus_queue.cc:1035] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 20:13:29.275360 12102 leader_election.cc:341] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:29.283316 12247 raft_consensus.cc:2454] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:29.297580 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:29.299427 12311 tablet_bootstrap.cc:492] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:29.299983 12311 ts_tablet_manager.cc:1397] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.457s user 0.035s sys 0.033s
I20260808 20:13:29.302418 12309 tablet_bootstrap.cc:654] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:29.309567 12311 raft_consensus.cc:348] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.310269 12311 raft_consensus.cc:374] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:29.310527 12311 raft_consensus.cc:729] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:29.311072 12311 consensus_queue.cc:260] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.323184 12311 ts_tablet_manager.cc:1428] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.023s user 0.003s sys 0.002s
I20260808 20:13:29.332629 12101 leader_election.cc:302] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:29.334024 12313 tablet_bootstrap.cc:654] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:29.336097 12311 tablet_bootstrap.cc:492] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:29.334362 12576 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:29.346709 12576 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.347182 12766 raft_consensus.cc:2781] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
W20260808 20:13:29.353623 12102 leader_election.cc:341] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
W20260808 20:13:29.362587 11966 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 20:13:29.363814 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:13:29.380111 11994 consensus_peers.cc:487] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 -> Peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Couldn't send request to peer b9d94f35ea15421aa1d721201c782d8a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:29.383188 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:29.348798 12576 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:29.389070 12766 raft_consensus.cc:686] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:29.390569 12766 consensus_queue.cc:237] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.401816 11896 catalog_manager.cc:5495] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:29.428812 12313 tablet_bootstrap.cc:492] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:29.429333 12313 ts_tablet_manager.cc:1397] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.173s user 0.008s sys 0.012s
I20260808 20:13:29.431824 12313 raft_consensus.cc:348] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.432487 12313 raft_consensus.cc:374] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:29.432730 12313 raft_consensus.cc:729] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:29.433329 12313 consensus_queue.cc:260] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.442406 12313 ts_tablet_manager.cc:1428] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
W20260808 20:13:29.463369 12205 leader_election.cc:341] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:29.464143 12205 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:29.465302 12389 raft_consensus.cc:2726] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:29.465742 12311 tablet_bootstrap.cc:654] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:29.486222 11894 catalog_manager.cc:5495] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
W20260808 20:13:29.492534 12800 log.cc:924] Time spent T 1d7d7a85e3964f3f9d5f2dc39e23340b P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.055s user 0.000s sys 0.013s
I20260808 20:13:29.494342 12309 tablet_bootstrap.cc:492] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:29.495499 12386 consensus_queue.cc:1035] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:29.501173 12309 ts_tablet_manager.cc:1397] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.430s user 0.038s sys 0.023s
I20260808 20:13:29.501890 12313 tablet_bootstrap.cc:492] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:29.506130 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:29.507416 12247 raft_consensus.cc:2379] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 1: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:13:29.510550 12309 raft_consensus.cc:348] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.512331 12309 raft_consensus.cc:374] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:29.512619 12309 raft_consensus.cc:729] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:29.513221 12309 consensus_queue.cc:260] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.550446 12309 ts_tablet_manager.cc:1428] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.049s user 0.005s sys 0.000s
I20260808 20:13:29.551244 12309 tablet_bootstrap.cc:492] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:29.523010 12101 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:29.549229 12150 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:29.566684 12573 raft_consensus.cc:2726] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:29.572567 12767 consensus_queue.cc:1035] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260808 20:13:29.575487 12311 tablet_bootstrap.cc:492] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:29.578454 12311 ts_tablet_manager.cc:1397] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.243s user 0.006s sys 0.016s
I20260808 20:13:29.642829 12313 tablet_bootstrap.cc:654] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:29.647574 12311 raft_consensus.cc:348] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.648274 12311 raft_consensus.cc:374] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:29.648557 12311 raft_consensus.cc:729] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:29.649199 12311 consensus_queue.cc:260] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.651347 12311 ts_tablet_manager.cc:1428] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.073s user 0.003s sys 0.003s
I20260808 20:13:29.652344 12311 tablet_bootstrap.cc:492] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:29.657943 12309 tablet_bootstrap.cc:654] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:29.659770 11897 catalog_manager.cc:5495] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:29.668483 12311 tablet_bootstrap.cc:654] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:29.671419 12389 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:29.672084 12389 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.674564 12389 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:29.676966 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:13:29.679489 12205 leader_election.cc:341] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:29.693905 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:13:29.695744 12206 leader_election.cc:341] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:29.696242 12206 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:29.697805 12766 consensus_queue.cc:1035] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:29.702338 12469 raft_consensus.cc:2726] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 20:13:29.712158 12102 consensus_peers.cc:487] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 -> Peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Couldn't send request to peer 4cd46f13bceb4c3c9ef71e755e17b995. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:29.714849 12472 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:29.715302 12472 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.715329 12313 tablet_bootstrap.cc:492] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:29.716425 12313 ts_tablet_manager.cc:1397] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.215s user 0.035s sys 0.027s
I20260808 20:13:29.719540 12313 raft_consensus.cc:348] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.720294 12313 raft_consensus.cc:374] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:29.720546 12313 raft_consensus.cc:729] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:29.721890 12313 consensus_queue.cc:260] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.726886 12313 ts_tablet_manager.cc:1428] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.010s user 0.003s sys 0.003s
I20260808 20:13:29.727739 12313 tablet_bootstrap.cc:492] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:29.728264 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:29.731966 12472 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:29.733741 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:29.734328 12251 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:29.729086 12142 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:29.735730 11994 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:29.739060 12386 raft_consensus.cc:2781] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:29.740090 12386 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:29.740518 12386 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:29.742918 12309 tablet_bootstrap.cc:492] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:29.742821 12311 tablet_bootstrap.cc:492] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:29.743873 12311 ts_tablet_manager.cc:1397] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.092s user 0.011s sys 0.011s
I20260808 20:13:29.743373 12309 ts_tablet_manager.cc:1397] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.192s user 0.017s sys 0.016s
I20260808 20:13:29.747399 12311 raft_consensus.cc:348] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.748039 12311 raft_consensus.cc:374] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:29.748302 12311 raft_consensus.cc:729] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:29.748263 12389 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:29.749042 12389 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.749222 12311 consensus_queue.cc:260] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.750973 12311 ts_tablet_manager.cc:1428] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260808 20:13:29.751760 12311 tablet_bootstrap.cc:492] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:29.751544 12389 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:29.755095 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:13:29.757457 12205 leader_election.cc:341] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:29.757337 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:13:29.758862 12206 leader_election.cc:341] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:29.759245 12206 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:29.760119 12389 raft_consensus.cc:2726] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:29.770077 12309 raft_consensus.cc:348] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.770566 12309 raft_consensus.cc:374] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:29.770767 12309 raft_consensus.cc:729] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:29.771404 12309 consensus_queue.cc:260] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.773097 12309 ts_tablet_manager.cc:1428] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20260808 20:13:29.773921 12309 tablet_bootstrap.cc:492] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:29.775506 12311 tablet_bootstrap.cc:654] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:29.778916 12313 tablet_bootstrap.cc:654] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:29.782193 12309 tablet_bootstrap.cc:654] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:29.786253 12386 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.793524 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:29.794229 12386 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:29.794315 12142 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:29.795039 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:29.795576 12251 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:29.801260 12311 tablet_bootstrap.cc:492] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:29.801694 12311 ts_tablet_manager.cc:1397] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.009s
I20260808 20:13:29.804826 12251 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:29.804998 12311 raft_consensus.cc:348] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.805758 12311 raft_consensus.cc:374] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:29.805756 12142 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:29.806291 12311 raft_consensus.cc:729] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:29.806948 12311 consensus_queue.cc:260] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.807044 11994 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:29.809224 12311 ts_tablet_manager.cc:1428] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20260808 20:13:29.809101 12770 raft_consensus.cc:2781] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:29.809795 12770 raft_consensus.cc:686] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:29.810179 12311 tablet_bootstrap.cc:492] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:29.810937 12770 consensus_queue.cc:237] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.820758 12309 tablet_bootstrap.cc:492] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:29.821204 12309 ts_tablet_manager.cc:1397] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.013s
I20260808 20:13:29.825690 12309 raft_consensus.cc:348] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.826404 12309 raft_consensus.cc:374] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:29.826699 12309 raft_consensus.cc:729] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:29.827334 12309 consensus_queue.cc:260] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.829038 12309 ts_tablet_manager.cc:1428] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260808 20:13:29.829974 12309 tablet_bootstrap.cc:492] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:29.831786 12311 tablet_bootstrap.cc:654] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:29.836032 11894 catalog_manager.cc:5495] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:29.838476 12309 tablet_bootstrap.cc:654] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:29.862680 12313 tablet_bootstrap.cc:492] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:29.863055 12313 ts_tablet_manager.cc:1397] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.135s user 0.031s sys 0.007s
I20260808 20:13:29.866134 12313 raft_consensus.cc:348] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.866787 12313 raft_consensus.cc:374] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:29.867039 12313 raft_consensus.cc:729] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:29.867595 12313 consensus_queue.cc:260] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.871301 12313 ts_tablet_manager.cc:1428] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260808 20:13:29.872803 12313 tablet_bootstrap.cc:492] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
W20260808 20:13:29.880165 12102 consensus_peers.cc:487] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 -> Peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Couldn't send request to peer 4cd46f13bceb4c3c9ef71e755e17b995. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:29.880765 12313 tablet_bootstrap.cc:654] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:29.882285 12311 tablet_bootstrap.cc:492] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:29.882738 12311 ts_tablet_manager.cc:1397] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.073s user 0.009s sys 0.015s
I20260808 20:13:29.884505 12769 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:29.885041 12769 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.886221 12311 raft_consensus.cc:348] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.886876 12311 raft_consensus.cc:374] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:29.887120 12311 raft_consensus.cc:729] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:29.887511 12769 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:29.887717 12311 consensus_queue.cc:260] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.888304 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:29.889217 12040 raft_consensus.cc:2454] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:29.889370 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:29.889976 12311 ts_tablet_manager.cc:1428] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260808 20:13:29.890528 12102 leader_election.cc:302] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:13:29.890769 12311 tablet_bootstrap.cc:492] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:29.891321 12769 raft_consensus.cc:2781] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:29.891638 12769 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:13:29.893507 12101 leader_election.cc:341] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:29.892169 12769 raft_consensus.cc:3037] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:29.899389 12309 tablet_bootstrap.cc:492] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:29.899803 12309 ts_tablet_manager.cc:1397] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.070s user 0.009s sys 0.013s
I20260808 20:13:29.903849 12767 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:29.904300 12767 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.904779 12311 tablet_bootstrap.cc:654] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:29.906610 12767 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:29.908954 12309 raft_consensus.cc:348] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.909087 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:29.912307 12245 raft_consensus.cc:2454] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:29.913692 11994 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:29.914888 12767 raft_consensus.cc:2781] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:29.915287 12767 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:29.915942 12767 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:29.910501 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:29.911830 12309 raft_consensus.cc:374] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:29.916992 12309 raft_consensus.cc:729] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
W20260808 20:13:29.917526 12000 leader_election.cc:341] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:29.917639 12309 consensus_queue.cc:260] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.920185 12309 ts_tablet_manager.cc:1428] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.020s user 0.003s sys 0.003s
I20260808 20:13:29.923957 12309 tablet_bootstrap.cc:492] T 1e0d98ab3b4945d8b845711f4ddd7153 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:29.927130 12769 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.935870 12578 raft_consensus.cc:482] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:29.936226 12767 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.936376 12578 raft_consensus.cc:504] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.939518 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:13:29.941707 12101 leader_election.cc:341] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:29.943302 12313 tablet_bootstrap.cc:492] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:29.943495 12767 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:29.943783 12313 ts_tablet_manager.cc:1397] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.071s user 0.013s sys 0.016s
I20260808 20:13:29.943783 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:29.944597 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:29.944948 12769 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:29.945205 12245 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:29.945516 12311 tablet_bootstrap.cc:492] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
W20260808 20:13:29.945816 12000 leader_election.cc:341] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:29.945941 12311 ts_tablet_manager.cc:1397] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.055s user 0.011s sys 0.007s
I20260808 20:13:29.947255 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:29.947877 12043 raft_consensus.cc:3037] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:29.944866 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:29.950500 12040 raft_consensus.cc:2454] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:29.953894 12102 leader_election.cc:302] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:29.952584 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:29.954888 12573 raft_consensus.cc:2781] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:29.953708 12578 leader_election.cc:288] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:13:29.956296 12101 leader_election.cc:341] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:29.955242 12573 raft_consensus.cc:482] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:29.956846 12573 raft_consensus.cc:3037] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:29.955565 12309 tablet_bootstrap.cc:654] T 1e0d98ab3b4945d8b845711f4ddd7153 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:29.956058 12311 raft_consensus.cc:348] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.958395 12311 raft_consensus.cc:374] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:29.958691 12311 raft_consensus.cc:729] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:29.956861 12576 consensus_queue.cc:1035] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260808 20:13:29.959503 12311 consensus_queue.cc:260] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.960338 12767 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:29.961037 12767 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.964658 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:29.971976 12245 raft_consensus.cc:2454] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:29.973935 11994 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:29.975643 12311 ts_tablet_manager.cc:1428] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20260808 20:13:29.976295 12311 tablet_bootstrap.cc:492] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:29.971899 12573 raft_consensus.cc:504] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.973822 12767 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:29.976084 12313 raft_consensus.cc:348] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.977106 12386 raft_consensus.cc:2781] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:29.979915 12313 raft_consensus.cc:374] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:29.980213 12313 raft_consensus.cc:729] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:29.980235 12386 raft_consensus.cc:686] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:29.980607 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:29.981164 12573 leader_election.cc:288] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:29.981026 12313 consensus_queue.cc:260] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.980952 12043 raft_consensus.cc:2454] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:29.982220 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:29.981586 12386 consensus_queue.cc:237] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.983632 12313 ts_tablet_manager.cc:1428] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.028s user 0.005s sys 0.001s
W20260808 20:13:29.984041 12101 leader_election.cc:341] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:29.985137 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:29.985303 12040 raft_consensus.cc:3037] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:29.985915 12102 leader_election.cc:302] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:29.987978 12313 tablet_bootstrap.cc:492] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:29.989176 12769 raft_consensus.cc:2781] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:29.989873 12769 raft_consensus.cc:686] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:29.990922 12769 consensus_queue.cc:237] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:29.995116 12469 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:29.995721 12469 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.001547 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:13:30.002928 12205 leader_election.cc:341] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:30.009061 12469 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:30.012468 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:30.045680 12040 raft_consensus.cc:2454] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
W20260808 20:13:30.052805 12206 leader_election.cc:341] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:30.053274 12206 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:30.056835 12761 raft_consensus.cc:2726] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:30.053961 11994 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:30.054818 11894 catalog_manager.cc:5495] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:30.063674 12102 leader_election.cc:302] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:30.058373 12770 raft_consensus.cc:2726] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:30.064941 12309 tablet_bootstrap.cc:492] T 1e0d98ab3b4945d8b845711f4ddd7153 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:30.065353 12309 ts_tablet_manager.cc:1397] T 1e0d98ab3b4945d8b845711f4ddd7153 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.143s user 0.018s sys 0.017s
I20260808 20:13:30.067909 12389 raft_consensus.cc:482] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:30.069528 12389 raft_consensus.cc:504] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.075845 12389 leader_election.cc:288] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:30.067503 12576 raft_consensus.cc:2781] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:30.078478 12576 raft_consensus.cc:686] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:30.079663 12576 consensus_queue.cc:237] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.072393 12309 raft_consensus.cc:348] T 1e0d98ab3b4945d8b845711f4ddd7153 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.080396 11897 catalog_manager.cc:5495] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:30.094216 12309 raft_consensus.cc:374] T 1e0d98ab3b4945d8b845711f4ddd7153 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:30.100329 12309 raft_consensus.cc:729] T 1e0d98ab3b4945d8b845711f4ddd7153 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:30.111596 12309 consensus_queue.cc:260] T 1e0d98ab3b4945d8b845711f4ddd7153 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.114810 12472 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:30.115267 12472 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.118965 12472 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:30.118981 12309 ts_tablet_manager.cc:1428] T 1e0d98ab3b4945d8b845711f4ddd7153 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.053s user 0.005s sys 0.003s
I20260808 20:13:30.121446 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:30.126472 12309 tablet_bootstrap.cc:492] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
W20260808 20:13:30.128180 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 46676.3us
I20260808 20:13:30.132563 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:30.133096 12311 tablet_bootstrap.cc:654] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:30.141531 12766 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:30.141992 12766 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.145196 12472 raft_consensus.cc:482] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:30.144169 12573 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:30.145839 12573 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.146711 12472 raft_consensus.cc:504] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.138854 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:30.149078 12472 leader_election.cc:288] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:30.150187 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:30.150578 12573 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:30.150738 12250 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:30.152470 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:30.152765 12766 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:30.152645 11994 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:30.153038 12250 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:30.154464 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:30.154953 12045 raft_consensus.cc:3037] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:30.158057 12205 leader_election.cc:302] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:13:30.158859 12000 leader_election.cc:341] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:30.159313 12469 raft_consensus.cc:2726] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:30.158138 12770 raft_consensus.cc:2781] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:30.160455 12770 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:30.160786 12770 raft_consensus.cc:3037] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:30.163481 12101 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:30.170141 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:30.170810 12250 raft_consensus.cc:2454] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:30.172559 12101 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:30.172814 12313 tablet_bootstrap.cc:654] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:30.173676 12766 raft_consensus.cc:2781] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:30.175530 12766 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:30.175895 12766 raft_consensus.cc:3037] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:30.176980 12309 tablet_bootstrap.cc:654] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:30.179658 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:30.174351 12855 raft_consensus.cc:2781] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:30.182619 12855 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:30.183049 12855 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:30.183383 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:30.182670 12770 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.188570 11894 catalog_manager.cc:5495] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:30.190313 12770 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:30.191131 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:30.191645 12250 raft_consensus.cc:3037] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:30.191951 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:30.194476 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:30.194356 12855 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.197548 12855 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:30.197598 12250 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
W20260808 20:13:30.198560 12102 leader_election.cc:341] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
W20260808 20:13:30.199668 12102 leader_election.cc:341] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:30.200448 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:30.200503 12000 leader_election.cc:302] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:30.201383 12248 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:30.202787 11994 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:30.205152 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:30.205623 12569 raft_consensus.cc:2781] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:30.206096 12569 raft_consensus.cc:686] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
W20260808 20:13:30.206522 12102 leader_election.cc:341] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:30.206840 12569 consensus_queue.cc:237] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:13:30.207939 12000 leader_election.cc:341] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:30.210361 12311 tablet_bootstrap.cc:492] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:30.210709 12311 ts_tablet_manager.cc:1397] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.235s user 0.020s sys 0.009s
I20260808 20:13:30.214944 12311 raft_consensus.cc:348] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.215425 12248 raft_consensus.cc:2454] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:30.215925 12311 raft_consensus.cc:374] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:30.216349 12311 raft_consensus.cc:729] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:30.214995 12766 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.217240 12311 consensus_queue.cc:260] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.218791 12766 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:30.219712 12311 ts_tablet_manager.cc:1428] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20260808 20:13:30.220623 12311 tablet_bootstrap.cc:492] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:30.221282 12309 tablet_bootstrap.cc:492] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:30.221136 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:30.221709 12309 ts_tablet_manager.cc:1397] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.095s user 0.011s sys 0.003s
I20260808 20:13:30.224713 12101 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:30.225023 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:30.225630 12766 raft_consensus.cc:2781] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:30.225761 12245 raft_consensus.cc:3037] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:30.227088 12766 raft_consensus.cc:686] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
W20260808 20:13:30.228796 12102 leader_election.cc:341] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:30.234905 12313 tablet_bootstrap.cc:492] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:30.235313 12313 ts_tablet_manager.cc:1397] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.248s user 0.014s sys 0.008s
I20260808 20:13:30.228232 12766 consensus_queue.cc:237] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.237475 12309 raft_consensus.cc:348] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.238188 12309 raft_consensus.cc:374] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:30.238438 12309 raft_consensus.cc:729] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:30.239413 12309 consensus_queue.cc:260] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.242376 12309 ts_tablet_manager.cc:1428] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.020s user 0.008s sys 0.000s
I20260808 20:13:30.245530 12309 tablet_bootstrap.cc:492] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:30.252045 12245 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:30.254313 12101 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:30.254406 12770 raft_consensus.cc:2726] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:30.255371 12578 raft_consensus.cc:2781] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:30.255941 12578 raft_consensus.cc:686] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:30.257148 12578 consensus_queue.cc:237] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.254223 12313 raft_consensus.cc:348] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.258703 12313 raft_consensus.cc:374] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:30.258975 12313 raft_consensus.cc:729] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:30.260701 12043 raft_consensus.cc:3037] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:30.262225 12311 tablet_bootstrap.cc:654] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:30.259649 12313 consensus_queue.cc:260] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.267102 12313 ts_tablet_manager.cc:1428] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.031s user 0.005s sys 0.000s
I20260808 20:13:30.273803 12313 tablet_bootstrap.cc:492] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:30.276376 12309 tablet_bootstrap.cc:654] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:30.306998 12250 raft_consensus.cc:3037] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:30.311232 12313 tablet_bootstrap.cc:654] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:30.306885 11897 catalog_manager.cc:5495] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:30.328089 12389 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:30.328634 12389 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.330544 12389 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:30.345150 11894 catalog_manager.cc:5495] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:30.347177 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:13:30.349074 12205 leader_election.cc:341] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:30.362960 12311 tablet_bootstrap.cc:492] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:30.363456 12311 ts_tablet_manager.cc:1397] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.143s user 0.015s sys 0.031s
I20260808 20:13:30.377481 12309 tablet_bootstrap.cc:492] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:30.377942 12309 ts_tablet_manager.cc:1397] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.133s user 0.015s sys 0.010s
I20260808 20:13:30.369844 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:13:30.385108 12206 leader_election.cc:341] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:30.385768 12206 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:30.386780 12852 raft_consensus.cc:2726] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:30.392620 12309 raft_consensus.cc:348] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.401366 12309 raft_consensus.cc:374] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:30.401650 12309 raft_consensus.cc:729] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:30.402256 12309 consensus_queue.cc:260] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:13:30.405294 12101 consensus_peers.cc:487] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 -> Peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Couldn't send request to peer b9d94f35ea15421aa1d721201c782d8a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:30.406533 12309 ts_tablet_manager.cc:1428] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20260808 20:13:30.407452 12309 tablet_bootstrap.cc:492] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:30.414472 11894 catalog_manager.cc:5495] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:30.417974 12472 consensus_queue.cc:1035] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:30.419258 12313 tablet_bootstrap.cc:492] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:30.419690 12313 ts_tablet_manager.cc:1397] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.146s user 0.026s sys 0.033s
I20260808 20:13:30.422009 12313 raft_consensus.cc:348] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.422646 12313 raft_consensus.cc:374] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:30.422909 12313 raft_consensus.cc:729] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:30.423557 12313 consensus_queue.cc:260] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.423748 12311 raft_consensus.cc:348] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.428769 12311 raft_consensus.cc:374] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:30.429136 12311 raft_consensus.cc:729] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:30.429780 12311 consensus_queue.cc:260] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.431882 12311 ts_tablet_manager.cc:1428] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.068s user 0.003s sys 0.003s
I20260808 20:13:30.446463 12386 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:30.446893 12386 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.448583 12386 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:30.452435 12767 raft_consensus.cc:482] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:30.452833 12767 raft_consensus.cc:504] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.452868 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:30.454869 12767 leader_election.cc:288] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:30.455854 12767 consensus_queue.cc:1035] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 20:13:30.452834 11966 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 20:13:30.457859 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:13:30.461443 12876 log.cc:924] Time spent T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.057s user 0.004s sys 0.000s
I20260808 20:13:30.469240 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:30.465214 12313 ts_tablet_manager.cc:1428] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.045s user 0.007s sys 0.014s
I20260808 20:13:30.470199 12245 raft_consensus.cc:2379] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 1: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:13:30.470954 12313 tablet_bootstrap.cc:492] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:30.471130 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:30.474012 12576 consensus_queue.cc:1035] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:30.473739 12309 tablet_bootstrap.cc:654] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:30.481628 12311 tablet_bootstrap.cc:492] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:30.481381 11994 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:30.483891 11994 leader_election.cc:302] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:30.485303 12569 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:30.488317 12386 raft_consensus.cc:2726] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:30.495440 12146 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:30.524984 12469 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:30.525565 12469 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.527400 12576 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:30.527860 12576 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.531028 12311 tablet_bootstrap.cc:654] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:30.531239 12576 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:30.532092 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:13:30.533275 12101 leader_election.cc:341] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:30.535773 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:13:30.537487 12205 leader_election.cc:341] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:30.537994 12469 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:30.533864 12576 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:30.538882 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:30.527129 12038 raft_consensus.cc:3037] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
W20260808 20:13:30.541628 12102 leader_election.cc:341] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:30.541993 12102 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:30.545451 12386 consensus_queue.cc:1035] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:30.545585 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:30.548211 12578 raft_consensus.cc:2726] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 20:13:30.549659 12206 leader_election.cc:341] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:30.549597 12472 consensus_queue.cc:1035] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:30.550141 12206 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:30.549970 12313 tablet_bootstrap.cc:654] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:30.556489 12469 raft_consensus.cc:2726] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:30.545114 12576 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.571920 12309 tablet_bootstrap.cc:492] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:30.572387 12309 ts_tablet_manager.cc:1397] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.165s user 0.003s sys 0.019s
I20260808 20:13:30.576855 12309 raft_consensus.cc:348] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.578475 12309 raft_consensus.cc:374] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:30.578744 12309 raft_consensus.cc:729] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:30.579679 12309 consensus_queue.cc:260] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.589111 12311 tablet_bootstrap.cc:492] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:30.592947 12311 ts_tablet_manager.cc:1397] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.111s user 0.016s sys 0.013s
I20260808 20:13:30.595739 12576 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:30.595703 12311 raft_consensus.cc:348] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.596410 12311 raft_consensus.cc:374] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:30.596699 12311 raft_consensus.cc:729] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:30.599058 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:30.599663 12250 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:30.601747 12101 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:30.602312 12309 ts_tablet_manager.cc:1428] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.030s user 0.000s sys 0.006s
I20260808 20:13:30.597287 12311 consensus_queue.cc:260] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.603721 12853 raft_consensus.cc:2781] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:30.611433 12853 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:30.611990 12853 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:30.612957 12569 raft_consensus.cc:2726] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 20:13:30.614995 12000 consensus_peers.cc:487] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 -> Peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Couldn't send request to peer 78b9263889cd4f1582578a3aa4f644d9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:30.624491 12309 tablet_bootstrap.cc:492] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:30.630883 12859 consensus_queue.cc:1035] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:30.633849 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:13:30.637223 12102 leader_election.cc:341] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:30.640064 12311 ts_tablet_manager.cc:1428] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.047s user 0.006s sys 0.001s
I20260808 20:13:30.641587 12311 tablet_bootstrap.cc:492] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:30.651937 12043 raft_consensus.cc:3037] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:30.658198 12576 consensus_queue.cc:1035] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:30.650403 12569 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:30.665300 12569 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.667277 12578 consensus_queue.cc:1035] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260808 20:13:30.671499 12101 consensus_peers.cc:487] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 -> Peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Couldn't send request to peer b9d94f35ea15421aa1d721201c782d8a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:30.681419 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:30.682055 12250 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:30.686908 12309 tablet_bootstrap.cc:654] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:30.691208 11994 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:30.682077 12569 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:30.684684 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:30.705276 12313 tablet_bootstrap.cc:492] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:30.705763 12313 ts_tablet_manager.cc:1397] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.235s user 0.042s sys 0.010s
I20260808 20:13:30.695271 12767 raft_consensus.cc:2781] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260808 20:13:30.713176 12000 leader_election.cc:341] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:30.715031 12311 tablet_bootstrap.cc:654] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:30.713936 12767 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:30.731038 12766 consensus_queue.cc:1035] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:30.739688 12313 raft_consensus.cc:348] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.740374 12313 raft_consensus.cc:374] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:30.740622 12313 raft_consensus.cc:729] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:30.731113 12569 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:30.745627 12569 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.733435 12767 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:30.758620 12573 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:30.763386 12569 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:30.741647 12313 consensus_queue.cc:260] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.764806 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:30.766304 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:30.766949 12313 ts_tablet_manager.cc:1428] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.061s user 0.007s sys 0.000s
I20260808 20:13:30.769687 12313 tablet_bootstrap.cc:492] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:30.760582 12573 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.771821 12573 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:30.773064 12000 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:30.779052 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:30.779673 12250 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:30.778239 12389 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:30.781385 12101 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:30.781697 12389 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.782560 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:13:30.785933 12102 leader_election.cc:341] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:30.796767 12472 raft_consensus.cc:2726] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:13:30.792788 12389 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:30.798316 12766 raft_consensus.cc:2781] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:30.801386 12766 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:30.801805 12766 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:30.802125 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:13:30.803696 12206 leader_election.cc:341] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:30.815703 12899 consensus_queue.cc:1035] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260808 20:13:30.816337 12853 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.834412 12313 tablet_bootstrap.cc:654] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:30.846290 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:30.849191 12853 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:13:30.851567 12102 leader_election.cc:341] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:30.824676 12767 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.816447 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:13:30.853698 12205 leader_election.cc:341] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:30.854089 12205 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:30.854918 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:30.856526 12389 raft_consensus.cc:2726] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 20:13:30.863052 12000 leader_election.cc:341] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:30.910017 12309 tablet_bootstrap.cc:492] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:30.910512 12309 ts_tablet_manager.cc:1397] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.286s user 0.023s sys 0.027s
I20260808 20:13:30.913064 12309 raft_consensus.cc:348] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.915750 12311 tablet_bootstrap.cc:492] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:30.916234 12311 ts_tablet_manager.cc:1397] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.275s user 0.008s sys 0.008s
I20260808 20:13:30.918394 12766 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.918193 12767 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:30.925082 12311 raft_consensus.cc:348] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.925802 12311 raft_consensus.cc:374] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:30.926046 12311 raft_consensus.cc:729] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:30.926638 12311 consensus_queue.cc:260] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.913703 12309 raft_consensus.cc:374] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:30.931206 12309 raft_consensus.cc:729] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:30.931730 12309 consensus_queue.cc:260] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.937062 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:30.937605 12250 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:30.940187 12311 ts_tablet_manager.cc:1428] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.024s user 0.003s sys 0.004s
I20260808 20:13:30.943260 12311 tablet_bootstrap.cc:492] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:30.947468 12250 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:30.948448 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:30.950454 12309 ts_tablet_manager.cc:1428] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.040s user 0.005s sys 0.001s
I20260808 20:13:30.951349 12309 tablet_bootstrap.cc:492] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
W20260808 20:13:30.951983 12102 leader_election.cc:341] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:30.949112 12766 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:13:30.960306 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 178981us
I20260808 20:13:30.966833 12313 tablet_bootstrap.cc:492] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:30.967324 12313 ts_tablet_manager.cc:1397] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.198s user 0.017s sys 0.016s
I20260808 20:13:30.970245 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:30.970839 12250 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:30.975620 12906 consensus_queue.cc:1035] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:30.980202 12309 tablet_bootstrap.cc:654] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
W20260808 20:13:30.983513 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 45596.2us
I20260808 20:13:30.984308 11994 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:30.985935 12767 raft_consensus.cc:2781] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:30.988356 12311 tablet_bootstrap.cc:654] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:30.988399 12767 raft_consensus.cc:686] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:30.990204 12767 consensus_queue.cc:237] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:30.994629 12250 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:30.999372 12386 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:30.999799 12386 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.002730 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:31.003469 12245 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:31.006749 12766 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:31.007349 12766 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.009784 12386 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:31.010028 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:31.010887 12152 raft_consensus.cc:2454] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:31.012549 12000 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:13:31.012552 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:13:31.014724 11994 leader_election.cc:341] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:31.014146 12770 raft_consensus.cc:2781] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:31.026422 12770 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:31.026961 12770 raft_consensus.cc:3037] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:31.029874 12766 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:31.030191 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:31.061008 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:31.061722 12250 raft_consensus.cc:2379] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 1: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:13:31.069619 12313 raft_consensus.cc:348] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.077303 12313 raft_consensus.cc:374] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:31.077582 12313 raft_consensus.cc:729] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:31.110128 12311 tablet_bootstrap.cc:492] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:31.110661 12311 ts_tablet_manager.cc:1397] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.168s user 0.017s sys 0.008s
I20260808 20:13:31.113734 12101 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:31.113696 12311 raft_consensus.cc:348] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.117928 12311 raft_consensus.cc:374] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:31.118261 12311 raft_consensus.cc:729] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:31.118829 12313 consensus_queue.cc:260] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.119006 11896 catalog_manager.cc:5495] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:31.120478 12311 consensus_queue.cc:260] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.132325 12101 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:31.119199 12906 raft_consensus.cc:2781] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:31.137501 12573 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:31.142369 12906 raft_consensus.cc:686] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:31.142843 12309 tablet_bootstrap.cc:492] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:31.143296 12906 consensus_queue.cc:237] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.143288 12309 ts_tablet_manager.cc:1397] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.192s user 0.034s sys 0.010s
I20260808 20:13:31.143477 12770 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.136864 12311 ts_tablet_manager.cc:1428] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.026s user 0.006s sys 0.001s
I20260808 20:13:31.146384 12311 tablet_bootstrap.cc:492] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:31.157953 12313 ts_tablet_manager.cc:1428] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.190s user 0.018s sys 0.007s
I20260808 20:13:31.158890 12313 tablet_bootstrap.cc:492] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:31.168114 12309 raft_consensus.cc:348] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.168756 12309 raft_consensus.cc:374] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:31.169056 12309 raft_consensus.cc:729] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:31.169282 12770 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:31.169677 12309 consensus_queue.cc:260] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.173972 12309 ts_tablet_manager.cc:1428] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.030s user 0.000s sys 0.005s
I20260808 20:13:31.174798 12309 tablet_bootstrap.cc:492] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:31.173628 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:31.175513 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
W20260808 20:13:31.176744 11994 leader_election.cc:341] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:31.176061 12150 raft_consensus.cc:3037] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
W20260808 20:13:31.179867 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 47622.9us
I20260808 20:13:31.212378 12150 raft_consensus.cc:2454] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:31.213879 12000 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:31.224169 12472 raft_consensus.cc:2781] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:31.224596 12472 raft_consensus.cc:686] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:31.225244 12472 consensus_queue.cc:237] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.177006 12245 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:31.240844 12311 tablet_bootstrap.cc:654] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:31.243742 12101 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:31.244455 12855 raft_consensus.cc:2781] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:31.246542 12855 raft_consensus.cc:686] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:31.241370 12573 raft_consensus.cc:2726] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:31.248247 12855 consensus_queue.cc:237] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.257210 12309 tablet_bootstrap.cc:654] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:31.264132 12313 tablet_bootstrap.cc:654] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:31.272392 12576 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:31.272986 12576 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.275784 12576 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:31.276820 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:31.277498 12248 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:31.279486 12101 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:31.280058 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:31.280341 12906 raft_consensus.cc:2781] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:31.280732 12906 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:31.281061 12906 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
W20260808 20:13:31.284399 12102 leader_election.cc:341] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:31.291069 12906 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.294080 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:13:31.295804 12102 leader_election.cc:341] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:31.303092 11894 catalog_manager.cc:5495] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:31.310997 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:31.311609 12248 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:31.317981 12311 tablet_bootstrap.cc:492] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:31.319897 12311 ts_tablet_manager.cc:1397] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.174s user 0.008s sys 0.009s
I20260808 20:13:31.319456 12309 tablet_bootstrap.cc:492] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:31.321234 12309 ts_tablet_manager.cc:1397] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.147s user 0.003s sys 0.013s
I20260808 20:13:31.325605 12309 raft_consensus.cc:348] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.322821 12311 raft_consensus.cc:348] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.326413 12309 raft_consensus.cc:374] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:31.327222 12309 raft_consensus.cc:729] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:31.327832 12309 consensus_queue.cc:260] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.328975 12311 raft_consensus.cc:374] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:31.329308 12311 raft_consensus.cc:729] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:31.329975 12311 consensus_queue.cc:260] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.326249 12906 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:31.319598 12248 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:31.331892 12311 ts_tablet_manager.cc:1428] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20260808 20:13:31.333238 12101 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:31.334322 12917 raft_consensus.cc:2781] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:31.334833 12917 raft_consensus.cc:686] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:31.335518 11896 catalog_manager.cc:5495] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:31.335800 12917 consensus_queue.cc:237] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.340332 12911 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:31.340850 12911 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.344748 11897 catalog_manager.cc:5495] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:31.345942 12309 ts_tablet_manager.cc:1428] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.022s user 0.003s sys 0.002s
I20260808 20:13:31.346781 12309 tablet_bootstrap.cc:492] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:31.347875 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:31.351534 12313 tablet_bootstrap.cc:492] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:31.352010 12313 ts_tablet_manager.cc:1397] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.193s user 0.006s sys 0.033s
I20260808 20:13:31.349462 12911 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:31.332736 12311 tablet_bootstrap.cc:492] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:31.354580 12313 raft_consensus.cc:348] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.355260 12313 raft_consensus.cc:374] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:31.355516 12313 raft_consensus.cc:729] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:31.356334 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:31.357800 12205 leader_election.cc:302] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:31.358907 12911 raft_consensus.cc:2726] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:31.362490 12311 tablet_bootstrap.cc:654] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:31.368621 12313 consensus_queue.cc:260] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.377944 12309 tablet_bootstrap.cc:654] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:31.378831 12313 ts_tablet_manager.cc:1428] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.027s user 0.004s sys 0.000s
I20260808 20:13:31.384819 12309 tablet_bootstrap.cc:492] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:31.385761 12309 ts_tablet_manager.cc:1397] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.003s
I20260808 20:13:31.388193 12313 tablet_bootstrap.cc:492] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:31.388010 12309 raft_consensus.cc:348] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.389750 12309 raft_consensus.cc:374] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:31.391477 12309 raft_consensus.cc:729] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:31.391225 12311 tablet_bootstrap.cc:492] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:31.392117 12311 ts_tablet_manager.cc:1397] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.006s
I20260808 20:13:31.394325 12311 raft_consensus.cc:348] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.395035 12311 raft_consensus.cc:374] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:31.395561 12311 raft_consensus.cc:729] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:31.397019 12311 consensus_queue.cc:260] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.399076 12311 ts_tablet_manager.cc:1428] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260808 20:13:31.395110 12313 tablet_bootstrap.cc:654] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:31.396060 12309 consensus_queue.cc:260] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.403028 12309 ts_tablet_manager.cc:1428] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.017s user 0.003s sys 0.001s
I20260808 20:13:31.403864 12309 tablet_bootstrap.cc:492] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:31.477027 12311 tablet_bootstrap.cc:492] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:31.477991 12899 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:31.478153 12309 tablet_bootstrap.cc:654] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:31.479350 12472 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:31.479795 12472 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.478514 12899 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.482407 12899 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:31.483449 12472 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:31.492165 12311 tablet_bootstrap.cc:654] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:31.495038 11896 catalog_manager.cc:5495] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:31.502807 12313 tablet_bootstrap.cc:492] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:31.503309 12313 ts_tablet_manager.cc:1397] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.115s user 0.004s sys 0.008s
I20260808 20:13:31.504375 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:31.505266 12244 raft_consensus.cc:2379] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 1: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:13:31.507894 12313 raft_consensus.cc:348] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.508594 12313 raft_consensus.cc:374] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:31.508844 12313 raft_consensus.cc:729] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:31.509466 12313 consensus_queue.cc:260] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.511514 12313 ts_tablet_manager.cc:1428] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260808 20:13:31.512383 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:31.513206 12248 raft_consensus.cc:2379] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 1: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:13:31.512395 12313 tablet_bootstrap.cc:492] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:31.504418 12311 tablet_bootstrap.cc:492] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:31.515105 12311 ts_tablet_manager.cc:1397] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.008s
I20260808 20:13:31.517325 12311 raft_consensus.cc:348] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.517962 12311 raft_consensus.cc:374] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:31.518225 12311 raft_consensus.cc:729] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:31.518791 12311 consensus_queue.cc:260] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.520318 12311 ts_tablet_manager.cc:1428] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260808 20:13:31.521137 12311 tablet_bootstrap.cc:492] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:31.526988 12313 tablet_bootstrap.cc:654] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:31.527285 12311 tablet_bootstrap.cc:654] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:31.539324 12311 tablet_bootstrap.cc:492] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:31.539763 12311 ts_tablet_manager.cc:1397] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.006s
I20260808 20:13:31.542275 12311 raft_consensus.cc:348] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.543004 12311 raft_consensus.cc:374] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:31.543259 12311 raft_consensus.cc:729] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:31.543835 12311 consensus_queue.cc:260] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.545950 12311 ts_tablet_manager.cc:1428] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 20:13:31.546969 12311 tablet_bootstrap.cc:492] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:31.552049 12311 tablet_bootstrap.cc:654] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:31.555411 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:31.556783 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:31.565742 12245 raft_consensus.cc:3037] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:31.566718 12311 tablet_bootstrap.cc:492] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:31.567217 12311 ts_tablet_manager.cc:1397] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.005s
I20260808 20:13:31.569523 12311 raft_consensus.cc:348] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.570187 12311 raft_consensus.cc:374] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:31.570480 12311 raft_consensus.cc:729] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
I20260808 20:13:31.571066 12311 consensus_queue.cc:260] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.573168 12311 ts_tablet_manager.cc:1428] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 20:13:31.575213 12311 tablet_bootstrap.cc:492] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995: Bootstrap starting.
I20260808 20:13:31.576776 12899 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:31.577266 12899 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.579044 12899 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:31.576828 12576 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:31.579706 12855 consensus_queue.cc:1035] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:31.579843 12576 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.581424 12576 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:31.584321 12311 tablet_bootstrap.cc:654] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:31.590011 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:31.590574 12251 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:31.592132 11994 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:31.592772 12472 raft_consensus.cc:2781] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:31.593093 12472 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:31.593372 12472 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:31.613860 12767 raft_consensus.cc:482] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:31.614341 12767 raft_consensus.cc:504] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.616217 12767 leader_election.cc:288] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:31.617283 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:31.619132 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:31.625299 12767 consensus_queue.cc:1035] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:31.627113 12251 raft_consensus.cc:3037] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
W20260808 20:13:31.636319 11994 leader_election.cc:341] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:31.641552 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:31.644445 12309 tablet_bootstrap.cc:492] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:31.644552 12313 tablet_bootstrap.cc:492] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:31.644914 12309 ts_tablet_manager.cc:1397] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.241s user 0.035s sys 0.043s
I20260808 20:13:31.645231 12313 ts_tablet_manager.cc:1397] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.133s user 0.006s sys 0.004s
I20260808 20:13:31.646750 12472 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.647507 12309 raft_consensus.cc:348] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.648164 12309 raft_consensus.cc:374] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:31.648412 12309 raft_consensus.cc:729] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:31.649156 12146 raft_consensus.cc:3037] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:31.648998 12309 consensus_queue.cc:260] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.656032 12309 ts_tablet_manager.cc:1428] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.011s user 0.002s sys 0.004s
I20260808 20:13:31.656859 12309 tablet_bootstrap.cc:492] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:31.662465 12309 tablet_bootstrap.cc:654] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:31.665560 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:31.668063 12000 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:31.669345 12569 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
W20260808 20:13:31.674515 11994 consensus_peers.cc:487] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 -> Peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Couldn't send request to peer b9d94f35ea15421aa1d721201c782d8a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:31.674492 12309 tablet_bootstrap.cc:492] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:31.676106 12309 ts_tablet_manager.cc:1397] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.001s
I20260808 20:13:31.676980 12569 raft_consensus.cc:2726] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:31.684150 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:31.685788 12000 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:31.687338 12386 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:31.689368 12309 raft_consensus.cc:348] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.693249 12309 raft_consensus.cc:374] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:31.693531 12309 raft_consensus.cc:729] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:31.695122 12309 consensus_queue.cc:260] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.696784 12386 raft_consensus.cc:2726] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:31.697759 12309 ts_tablet_manager.cc:1428] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.021s user 0.006s sys 0.001s
I20260808 20:13:31.698704 12309 tablet_bootstrap.cc:492] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9: Bootstrap starting.
I20260808 20:13:31.704838 12472 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:31.705057 12309 tablet_bootstrap.cc:654] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:31.705852 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:31.706521 12248 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:31.715137 12248 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:31.716512 11994 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:31.719353 12102 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:31.723366 12855 consensus_queue.cc:1035] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:31.725122 12311 tablet_bootstrap.cc:492] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995: No bootstrap required, opened a new log
I20260808 20:13:31.725704 12311 ts_tablet_manager.cc:1397] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent bootstrapping tablet: real 0.151s user 0.038s sys 0.048s
I20260808 20:13:31.725844 12576 consensus_queue.cc:1035] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:31.726815 12386 raft_consensus.cc:2781] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:31.727254 12386 raft_consensus.cc:686] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:31.727916 12386 consensus_queue.cc:237] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.730609 12770 consensus_queue.cc:1035] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:31.731913 12311 raft_consensus.cc:348] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.735080 12311 raft_consensus.cc:374] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:31.735679 12311 raft_consensus.cc:729] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Initialized, Role: FOLLOWER
W20260808 20:13:31.737409 12000 leader_election.cc:341] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:31.738334 12311 consensus_queue.cc:260] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:13:31.744164 12000 leader_election.cc:341] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:31.745522 12000 leader_election.cc:302] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:31.751796 12472 raft_consensus.cc:2726] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:31.754750 12313 raft_consensus.cc:348] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.755610 12313 raft_consensus.cc:374] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:31.755921 12313 raft_consensus.cc:729] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:31.755786 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:31.757326 12150 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:31.756664 12313 consensus_queue.cc:260] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.765630 12311 ts_tablet_manager.cc:1428] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent starting tablet: real 0.039s user 0.006s sys 0.000s
I20260808 20:13:31.754654 12855 raft_consensus.cc:482] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:31.781487 12313 ts_tablet_manager.cc:1428] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.136s user 0.006s sys 0.000s
I20260808 20:13:31.800673 12313 tablet_bootstrap.cc:492] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:31.775844 12855 raft_consensus.cc:504] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.813266 12309 tablet_bootstrap.cc:492] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9: No bootstrap required, opened a new log
I20260808 20:13:31.813726 12309 ts_tablet_manager.cc:1397] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9: Time spent bootstrapping tablet: real 0.115s user 0.007s sys 0.009s
I20260808 20:13:31.816789 12150 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
W20260808 20:13:31.825362 11966 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 20:13:31.830205 12855 leader_election.cc:288] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:31.849117 12309 raft_consensus.cc:348] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.849858 12309 raft_consensus.cc:374] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:31.850132 12309 raft_consensus.cc:729] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Initialized, Role: FOLLOWER
I20260808 20:13:31.831431 12573 raft_consensus.cc:2726] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:31.861513 12309 consensus_queue.cc:260] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.898028 12309 ts_tablet_manager.cc:1428] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9: Time spent starting tablet: real 0.084s user 0.003s sys 0.003s
W20260808 20:13:31.899642 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 65914.2us
I20260808 20:13:31.874298 11894 catalog_manager.cc:5495] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:31.875828 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:31.848296 12569 consensus_queue.cc:1035] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
W20260808 20:13:31.923216 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 90995.6us
I20260808 20:13:31.918193 12040 raft_consensus.cc:2454] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
W20260808 20:13:31.924858 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 91516us
I20260808 20:13:31.935746 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:31.936373 12245 raft_consensus.cc:2454] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 0.
I20260808 20:13:31.948441 12472 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:31.948832 12472 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.950773 12101 leader_election.cc:302] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:31.954898 12472 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:31.955765 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:13:31.963675 11994 leader_election.cc:341] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Illegal state: must be running to vote when last-logged opid is not known
I20260808 20:13:31.965154 12855 raft_consensus.cc:2781] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:31.965540 12855 raft_consensus.cc:482] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:31.965847 12855 raft_consensus.cc:3037] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:31.968907 12472 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:31.969307 12472 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.973423 12472 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:31.975167 12313 tablet_bootstrap.cc:654] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:31.983678 12038 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:31.987340 12855 raft_consensus.cc:504] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:31.993673 12855 leader_election.cc:288] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:32.001268 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:13:32.002013 12249 raft_consensus.cc:2379] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 1: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:13:32.004868 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:32.006076 12248 raft_consensus.cc:3037] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:32.008095 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:32.008352 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:32.008658 12152 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 0.
I20260808 20:13:32.011539 12980 consensus_queue.cc:1035] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:32.014782 12248 raft_consensus.cc:2454] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:32.063844 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:32.065840 12038 raft_consensus.cc:3037] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:32.077286 12389 raft_consensus.cc:482] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:13:32.078346 12389 raft_consensus.cc:504] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:32.080317 12101 leader_election.cc:302] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:32.080611 12917 consensus_queue.cc:1035] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:32.084427 12389 leader_election.cc:288] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:32.097776 12313 tablet_bootstrap.cc:492] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:32.098696 12313 ts_tablet_manager.cc:1397] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.300s user 0.020s sys 0.000s
I20260808 20:13:32.101048 12576 consensus_queue.cc:1035] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:32.103868 12000 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:32.108424 12000 leader_election.cc:302] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:32.110237 12899 raft_consensus.cc:2781] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:32.110630 12899 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:32.110940 12899 raft_consensus.cc:3037] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:32.115906 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:32.116756 12152 raft_consensus.cc:2454] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 0.
I20260808 20:13:32.120151 12853 raft_consensus.cc:2781] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:32.120663 12853 raft_consensus.cc:686] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:32.108605 12770 raft_consensus.cc:2726] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 20:13:32.124604 12853 consensus_queue.cc:237] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:32.161630 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:32.162680 12045 raft_consensus.cc:2454] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 0.
I20260808 20:13:32.167157 12205 leader_election.cc:302] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:32.168565 12389 raft_consensus.cc:2781] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 20:13:32.168996 12389 raft_consensus.cc:482] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:13:32.169358 12389 raft_consensus.cc:3037] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:32.203274 12038 raft_consensus.cc:2454] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:13:32.185720 12313 raft_consensus.cc:348] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:32.214308 12313 raft_consensus.cc:374] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:32.214661 12313 raft_consensus.cc:729] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:32.214632 12389 raft_consensus.cc:504] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:32.215559 12313 consensus_queue.cc:260] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:32.216451 12389 leader_election.cc:288] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:32.206385 12899 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:32.218542 12313 ts_tablet_manager.cc:1428] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.119s user 0.008s sys 0.001s
I20260808 20:13:32.221881 12313 tablet_bootstrap.cc:492] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a: Bootstrap starting.
I20260808 20:13:32.229974 12971 consensus_queue.cc:1035] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:32.236340 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:32.236932 12152 raft_consensus.cc:3037] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:32.245635 12152 raft_consensus.cc:2454] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:13:32.229499 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:32.250236 12038 raft_consensus.cc:3037] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:32.258854 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:32.259479 12152 raft_consensus.cc:3037] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:32.259982 12038 raft_consensus.cc:2454] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:13:32.261349 12205 leader_election.cc:302] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:32.262255 12389 raft_consensus.cc:2781] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:32.262828 12389 raft_consensus.cc:686] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:32.263682 12389 consensus_queue.cc:237] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:32.258741 12899 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:32.279508 12899 consensus_queue.cc:1035] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:32.284004 12152 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:32.286842 12313 tablet_bootstrap.cc:654] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a: Neither blocks nor log segments found. Creating new log.
I20260808 20:13:32.289088 12000 leader_election.cc:302] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:13:32.291934 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:32.292502 12248 raft_consensus.cc:3037] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:32.290637 12899 raft_consensus.cc:2781] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 20:13:32.314610 12899 raft_consensus.cc:686] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:32.318972 12899 consensus_queue.cc:237] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:32.345201 12248 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:13:32.352053 11896 catalog_manager.cc:5495] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:32.362489 12313 tablet_bootstrap.cc:492] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a: No bootstrap required, opened a new log
I20260808 20:13:32.362975 12313 ts_tablet_manager.cc:1397] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a: Time spent bootstrapping tablet: real 0.141s user 0.011s sys 0.002s
I20260808 20:13:32.365425 12313 raft_consensus.cc:348] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:32.366065 12313 raft_consensus.cc:374] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 20:13:32.366299 12313 raft_consensus.cc:729] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Initialized, Role: FOLLOWER
I20260808 20:13:32.366882 12313 consensus_queue.cc:260] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:32.368942 12313 ts_tablet_manager.cc:1428] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 20:13:32.378022 11897 catalog_manager.cc:5495] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 0 to 1, leader changed from <none> to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:13:32.392810 11897 catalog_manager.cc:5495] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:32.613430 12248 raft_consensus.cc:3037] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 20:13:32.625012 12977 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:13:32.625581 12977 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:32.629878 12977 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:32.631731 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:13:32.633172 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:13:32.634487 12205 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:32.635200 12977 raft_consensus.cc:2726] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:13:32.666643 12906 consensus_queue.cc:1035] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 20:13:32.679318 12917 consensus_queue.cc:1035] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:32.690589 12389 consensus_queue.cc:1035] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:32.711876 12389 consensus_queue.cc:1035] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:32.821969 10336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 20:13:32.829080 13004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 20:13:32.850546 13006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 20:13:32.850579 13008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 20:13:32.852737 10336 server_base.cc:1034] running on GCE node
I20260808 20:13:32.854089 10336 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 20:13:32.854391 10336 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 20:13:32.854602 10336 hybrid_clock.cc:648] HybridClock initialized: now 1786220012854587 us; error 0 us; skew 500 ppm
I20260808 20:13:32.855333 10336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 20:13:32.875249 10336 webserver.cc:460] Webserver started at http://127.10.24.4:46239/ using document root <none> and password file <none>
I20260808 20:13:32.877731 10336 fs_manager.cc:359] Metadata directory not provided
I20260808 20:13:32.878103 10336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 20:13:32.878594 10336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 20:13:32.880354 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-3-root/instance:
uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
format_stamp: "Formatted at 2026-08-08 20:13:32 on dist-test-slave-ncp9"
I20260808 20:13:32.893431 10336 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.007s sys 0.005s
I20260808 20:13:32.897097 13013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 20:13:32.897960 10336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 20:13:32.898236 10336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-3-root
uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
format_stamp: "Formatted at 2026-08-08 20:13:32 on dist-test-slave-ncp9"
I20260808 20:13:32.898505 10336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 20:13:32.910641 12770 consensus_queue.cc:1035] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:32.915776 10336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 20:13:32.918022 10336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 20:13:32.921401 10336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 20:13:32.928423 10336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 20:13:32.928624 10336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 20:13:32.929013 10336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 20:13:32.929261 10336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 20:13:32.935936 12971 consensus_queue.cc:1035] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 20:13:33.024649 10336 rpc_server.cc:307] RPC server started. Bound to: 127.10.24.4:36901
I20260808 20:13:33.025134 13118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.24.4:36901 every 8 connection(s)
I20260808 20:13:33.030591 10336 auto_leader_rebalancer-test.cc:413] add a tserver: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:33.045957 13119 heartbeater.cc:347] Connected to a master server at 127.10.24.62:36417
I20260808 20:13:33.046307 13119 heartbeater.cc:464] Registering TS with master...
I20260808 20:13:33.046948 13119 heartbeater.cc:511] Master 127.10.24.62:36417 requested a full tablet report, sending...
I20260808 20:13:33.048446 11897 ts_manager.cc:194] Registered new tserver with Master: e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:13:33.049886 11897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44508
I20260808 20:13:33.968384 12038 consensus_queue.cc:237] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:33.974677 12140 raft_consensus.cc:1261] T 0be6c610fe68472b8bcb6232b862dbc1 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:33.975394 12244 raft_consensus.cc:1261] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:33.976738 12971 consensus_queue.cc:1035] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:33.982254 12971 consensus_queue.cc:1035] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:33.988344 12985 raft_consensus.cc:2932] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:33.996387 12244 raft_consensus.cc:2932] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:34.007594 11897 catalog_manager.cc:5495] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260808 20:13:34.013026 12001 consensus_peers.cc:487] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0be6c610fe68472b8bcb6232b862dbc1. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:34.019771 12140 raft_consensus.cc:2932] T 0be6c610fe68472b8bcb6232b862dbc1 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:34.046381 12147 consensus_queue.cc:237] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:34.052815 13119 heartbeater.cc:503] Master 127.10.24.62:36417 was elected leader, sending a full tablet report...
I20260808 20:13:34.053890 12244 raft_consensus.cc:1261] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:34.055547 12853 consensus_queue.cc:1035] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:34.061831 12038 raft_consensus.cc:1261] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:34.065930 13126 consensus_queue.cc:1035] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 20:13:34.076258 12853 raft_consensus.cc:2932] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:34.077816 12251 raft_consensus.cc:2932] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
W20260808 20:13:34.094089 12105 consensus_peers.cc:487] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d7d7a85e3964f3f9d5f2dc39e23340b. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:34.095438 11896 catalog_manager.cc:5495] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:34.100870 12140 consensus_queue.cc:237] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:34.097548 12043 raft_consensus.cc:2932] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:34.109354 12038 raft_consensus.cc:1261] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:34.109721 12251 raft_consensus.cc:1261] T f4918de0ffe94198a66ac1975ddbc693 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:34.110904 12853 consensus_queue.cc:1035] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260808 20:13:34.114115 12105 consensus_peers.cc:487] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4918de0ffe94198a66ac1975ddbc693. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:34.118844 12853 consensus_queue.cc:1035] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:34.119921 13151 raft_consensus.cc:2932] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:34.130419 12244 raft_consensus.cc:2932] T f4918de0ffe94198a66ac1975ddbc693 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:34.137531 12043 raft_consensus.cc:2932] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:34.140938 12140 consensus_queue.cc:237] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:34.147390 12251 raft_consensus.cc:1261] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:34.148802 12853 consensus_queue.cc:1035] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:34.150164 12038 raft_consensus.cc:1261] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:34.153106 13126 consensus_queue.cc:1035] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:34.175933 11894 catalog_manager.cc:5495] T f4918de0ffe94198a66ac1975ddbc693 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:34.189918 12853 raft_consensus.cc:2932] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:34.191238 12248 raft_consensus.cc:2932] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
W20260808 20:13:34.192761 12105 consensus_peers.cc:487] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e256c29dec964f49856cf204f950793b. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:34.202479 12038 raft_consensus.cc:2932] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:34.217047 11896 catalog_manager.cc:5495] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:34.504884 13167 ts_tablet_manager.cc:927] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:13:34.507302 13167 tablet_copy_client.cc:316] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.1:40747
I20260808 20:13:34.524135 12062 tablet_copy_service.cc:140] P 4cd46f13bceb4c3c9ef71e755e17b995: Received BeginTabletCopySession request for tablet 0be6c610fe68472b8bcb6232b862dbc1 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:49226)
I20260808 20:13:34.524601 12062 tablet_copy_service.cc:161] P 4cd46f13bceb4c3c9ef71e755e17b995: Beginning new tablet copy session on tablet 0be6c610fe68472b8bcb6232b862dbc1 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:49226: session id = e5bd9a4616ce4acfa20dc7a002984a20-0be6c610fe68472b8bcb6232b862dbc1
I20260808 20:13:34.531148 12062 tablet_copy_source_session.cc:205] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:34.536464 13167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be6c610fe68472b8bcb6232b862dbc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:34.549283 13167 tablet_copy_client.cc:791] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:34.549889 13167 tablet_copy_client.cc:655] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:34.553550 13167 tablet_copy_client.cc:531] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:34.561160 13167 tablet_bootstrap.cc:492] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:34.585184 13167 tablet_bootstrap.cc:492] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:34.585947 13167 tablet_bootstrap.cc:492] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:34.586432 13167 ts_tablet_manager.cc:1397] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.004s
I20260808 20:13:34.588716 13167 raft_consensus.cc:348] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:34.589591 13167 raft_consensus.cc:729] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:34.590186 13167 consensus_queue.cc:260] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:34.592697 13167 ts_tablet_manager.cc:1428] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 20:13:34.594272 12062 tablet_copy_service.cc:342] P 4cd46f13bceb4c3c9ef71e755e17b995: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-0be6c610fe68472b8bcb6232b862dbc1 received from {username='slave'} at 127.0.0.1:49226
I20260808 20:13:34.594668 12062 tablet_copy_service.cc:433] P 4cd46f13bceb4c3c9ef71e755e17b995: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-0be6c610fe68472b8bcb6232b862dbc1 on tablet 0be6c610fe68472b8bcb6232b862dbc1 with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:34.663815 13167 ts_tablet_manager.cc:927] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:34.666458 13167 tablet_copy_client.cc:316] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.2:35139
I20260808 20:13:34.675791 13178 ts_tablet_manager.cc:927] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:34.677325 13178 tablet_copy_client.cc:316] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.2:35139
I20260808 20:13:34.680729 12164 tablet_copy_service.cc:140] P 78b9263889cd4f1582578a3aa4f644d9: Received BeginTabletCopySession request for tablet e256c29dec964f49856cf204f950793b from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:46680)
I20260808 20:13:34.680955 12163 tablet_copy_service.cc:140] P 78b9263889cd4f1582578a3aa4f644d9: Received BeginTabletCopySession request for tablet 1d7d7a85e3964f3f9d5f2dc39e23340b from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:46680)
I20260808 20:13:34.681391 12164 tablet_copy_service.cc:161] P 78b9263889cd4f1582578a3aa4f644d9: Beginning new tablet copy session on tablet e256c29dec964f49856cf204f950793b from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:46680: session id = e5bd9a4616ce4acfa20dc7a002984a20-e256c29dec964f49856cf204f950793b
I20260808 20:13:34.682425 12163 tablet_copy_service.cc:161] P 78b9263889cd4f1582578a3aa4f644d9: Beginning new tablet copy session on tablet 1d7d7a85e3964f3f9d5f2dc39e23340b from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:46680: session id = e5bd9a4616ce4acfa20dc7a002984a20-1d7d7a85e3964f3f9d5f2dc39e23340b
I20260808 20:13:34.689783 12164 tablet_copy_source_session.cc:205] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:34.691176 12163 tablet_copy_source_session.cc:205] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:34.693662 13167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e256c29dec964f49856cf204f950793b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:34.695617 13178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7d7a85e3964f3f9d5f2dc39e23340b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:34.709777 13178 tablet_copy_client.cc:791] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:34.709978 13167 tablet_copy_client.cc:791] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:34.710440 13178 tablet_copy_client.cc:655] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:34.710593 13167 tablet_copy_client.cc:655] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:34.714020 13178 tablet_copy_client.cc:531] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:34.714829 13167 tablet_copy_client.cc:531] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:34.722031 13180 ts_tablet_manager.cc:927] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:34.722738 13178 tablet_bootstrap.cc:492] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:34.723789 13167 tablet_bootstrap.cc:492] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:34.734959 13180 tablet_copy_client.cc:316] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.2:35139
I20260808 20:13:34.736585 12164 tablet_copy_service.cc:140] P 78b9263889cd4f1582578a3aa4f644d9: Received BeginTabletCopySession request for tablet f4918de0ffe94198a66ac1975ddbc693 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:46680)
I20260808 20:13:34.737085 12164 tablet_copy_service.cc:161] P 78b9263889cd4f1582578a3aa4f644d9: Beginning new tablet copy session on tablet f4918de0ffe94198a66ac1975ddbc693 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:46680: session id = e5bd9a4616ce4acfa20dc7a002984a20-f4918de0ffe94198a66ac1975ddbc693
I20260808 20:13:34.748070 12164 tablet_copy_source_session.cc:205] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:34.753098 13180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4918de0ffe94198a66ac1975ddbc693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:34.785208 13178 tablet_bootstrap.cc:492] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:34.786185 13178 tablet_bootstrap.cc:492] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:34.786880 13178 ts_tablet_manager.cc:1397] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.064s user 0.009s sys 0.018s
I20260808 20:13:34.793004 13180 tablet_copy_client.cc:791] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:34.793535 13180 tablet_copy_client.cc:655] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:34.796847 13180 tablet_copy_client.cc:531] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:34.801736 13167 tablet_bootstrap.cc:492] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:34.802801 13167 tablet_bootstrap.cc:492] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:34.803521 13167 ts_tablet_manager.cc:1397] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.080s user 0.037s sys 0.031s
I20260808 20:13:34.805159 13180 tablet_bootstrap.cc:492] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:34.809521 13167 raft_consensus.cc:348] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:34.810396 13167 raft_consensus.cc:729] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:34.811100 13167 consensus_queue.cc:260] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:34.813733 13167 ts_tablet_manager.cc:1428] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260808 20:13:34.815281 12164 tablet_copy_service.cc:342] P 78b9263889cd4f1582578a3aa4f644d9: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-e256c29dec964f49856cf204f950793b received from {username='slave'} at 127.0.0.1:46680
I20260808 20:13:34.815670 12164 tablet_copy_service.cc:433] P 78b9263889cd4f1582578a3aa4f644d9: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-e256c29dec964f49856cf204f950793b on tablet e256c29dec964f49856cf204f950793b with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:34.815356 13178 raft_consensus.cc:348] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:34.818224 13178 raft_consensus.cc:729] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:34.818869 13178 consensus_queue.cc:260] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:34.821753 13178 ts_tablet_manager.cc:1428] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.035s user 0.006s sys 0.002s
I20260808 20:13:34.823164 12164 tablet_copy_service.cc:342] P 78b9263889cd4f1582578a3aa4f644d9: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-1d7d7a85e3964f3f9d5f2dc39e23340b received from {username='slave'} at 127.0.0.1:46680
I20260808 20:13:34.823503 12164 tablet_copy_service.cc:433] P 78b9263889cd4f1582578a3aa4f644d9: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-1d7d7a85e3964f3f9d5f2dc39e23340b on tablet 1d7d7a85e3964f3f9d5f2dc39e23340b with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:34.831591 13180 tablet_bootstrap.cc:492] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:34.832240 13180 tablet_bootstrap.cc:492] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:34.832851 13180 ts_tablet_manager.cc:1397] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.004s
I20260808 20:13:34.834906 13180 raft_consensus.cc:348] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:34.835593 13180 raft_consensus.cc:729] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:34.836123 13180 consensus_queue.cc:260] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:34.837841 13180 ts_tablet_manager.cc:1428] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260808 20:13:34.839146 12164 tablet_copy_service.cc:342] P 78b9263889cd4f1582578a3aa4f644d9: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-f4918de0ffe94198a66ac1975ddbc693 received from {username='slave'} at 127.0.0.1:46680
I20260808 20:13:34.839516 12164 tablet_copy_service.cc:433] P 78b9263889cd4f1582578a3aa4f644d9: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-f4918de0ffe94198a66ac1975ddbc693 on tablet f4918de0ffe94198a66ac1975ddbc693 with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:34.905834 13080 raft_consensus.cc:1203] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:35.092715 13080 raft_consensus.cc:1203] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:35.158370 13080 raft_consensus.cc:1203] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:35.235704 13080 raft_consensus.cc:1203] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:35.532254 13186 raft_consensus.cc:1050] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:35.549324 13186 consensus_queue.cc:237] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:35.558856 12147 raft_consensus.cc:1261] T 0be6c610fe68472b8bcb6232b862dbc1 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:35.558856 12248 raft_consensus.cc:1261] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:35.560098 13080 raft_consensus.cc:1261] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:35.560288 13189 consensus_queue.cc:1035] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:13:35.561367 12971 consensus_queue.cc:1035] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:35.563668 13186 consensus_queue.cc:1035] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:35.571326 13189 raft_consensus.cc:2932] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.577333 13080 raft_consensus.cc:2932] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.584280 12248 raft_consensus.cc:2932] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.595206 12147 raft_consensus.cc:2932] T 0be6c610fe68472b8bcb6232b862dbc1 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.592794 11897 catalog_manager.cc:5495] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.606302 12853 raft_consensus.cc:1050] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:35.608413 12853 consensus_queue.cc:237] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:35.623164 12248 raft_consensus.cc:1261] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:35.623348 13080 raft_consensus.cc:1261] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:35.625260 12038 raft_consensus.cc:1261] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:35.624620 13157 consensus_queue.cc:1035] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260808 20:13:35.627071 13158 consensus_queue.cc:1035] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:35.629204 13158 consensus_queue.cc:1035] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:13:35.655826 13157 raft_consensus.cc:2932] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.660169 12038 raft_consensus.cc:2932] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.659845 12248 raft_consensus.cc:2932] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.679407 13080 raft_consensus.cc:2932] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.692751 12853 raft_consensus.cc:1050] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:35.696113 12853 consensus_queue.cc:237] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:35.694105 13158 raft_consensus.cc:1050] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:35.700294 13158 consensus_queue.cc:237] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:35.700629 12038 consensus_queue.cc:237] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:35.701231 11896 catalog_manager.cc:5495] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:35.713737 12244 raft_consensus.cc:1261] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:35.716811 13186 consensus_queue.cc:1035] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:35.726356 13080 raft_consensus.cc:1261] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:35.728600 12770 consensus_queue.cc:1035] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:35.731449 12971 raft_consensus.cc:2932] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.732883 13079 raft_consensus.cc:1261] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:35.734557 12248 raft_consensus.cc:2932] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.732864 12147 consensus_queue.cc:237] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:35.737422 12038 raft_consensus.cc:1261] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:35.745509 11873 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 0be6c610fe68472b8bcb6232b862dbc1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:35.734540 13212 consensus_queue.cc:1035] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20260808 20:13:35.756292 11894 catalog_manager.cc:5495] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.763145 13078 raft_consensus.cc:2932] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.769166 13211 consensus_queue.cc:1035] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:13:35.787639 13080 raft_consensus.cc:1261] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20260808 20:13:35.774063 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 57646.5us
I20260808 20:13:35.795794 12131 tablet_service.cc:1514] Processing DeleteTablet for tablet 0be6c610fe68472b8bcb6232b862dbc1 with delete_type TABLET_DATA_TOMBSTONED (TS 78b9263889cd4f1582578a3aa4f644d9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53148
I20260808 20:13:35.798271 12244 raft_consensus.cc:1261] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:35.798715 12248 raft_consensus.cc:1261] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:35.799950 12248 raft_consensus.cc:1261] T f4918de0ffe94198a66ac1975ddbc693 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:35.792294 13222 consensus_queue.cc:1035] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 20:13:35.813468 13158 consensus_queue.cc:1035] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:35.814219 12853 consensus_queue.cc:1035] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:13:35.814241 13220 consensus_queue.cc:1035] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:13:35.821218 13079 raft_consensus.cc:1261] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:35.823432 12038 raft_consensus.cc:1261] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:35.823484 13225 consensus_queue.cc:1035] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:13:35.831928 13158 consensus_queue.cc:1035] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:35.833648 13158 raft_consensus.cc:2932] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.845548 13226 tablet_replica.cc:330] T 0be6c610fe68472b8bcb6232b862dbc1 P 78b9263889cd4f1582578a3aa4f644d9: stopping tablet replica
I20260808 20:13:35.848369 13226 raft_consensus.cc:2229] T 0be6c610fe68472b8bcb6232b862dbc1 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:35.848986 13226 raft_consensus.cc:2258] T 0be6c610fe68472b8bcb6232b862dbc1 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:35.842403 12251 raft_consensus.cc:2932] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.838424 13080 raft_consensus.cc:2932] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.867079 13226 ts_tablet_manager.cc:1905] T 0be6c610fe68472b8bcb6232b862dbc1 P 78b9263889cd4f1582578a3aa4f644d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:35.893451 11896 catalog_manager.cc:5495] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.897828 13157 raft_consensus.cc:2932] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.906581 12022 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d7d7a85e3964f3f9d5f2dc39e23340b with delete_type TABLET_DATA_TOMBSTONED (TS 4cd46f13bceb4c3c9ef71e755e17b995 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53014
I20260808 20:13:35.915460 12248 raft_consensus.cc:2932] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.914676 12043 raft_consensus.cc:2932] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.920509 11875 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 1d7d7a85e3964f3f9d5f2dc39e23340b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:35.922461 12038 raft_consensus.cc:2932] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.938473 13211 raft_consensus.cc:2932] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.944504 13236 tablet_replica.cc:330] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 4cd46f13bceb4c3c9ef71e755e17b995: stopping tablet replica
I20260808 20:13:35.948590 13226 ts_tablet_manager.cc:1918] T 0be6c610fe68472b8bcb6232b862dbc1 P 78b9263889cd4f1582578a3aa4f644d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:35.957679 13226 log.cc:1196] T 0be6c610fe68472b8bcb6232b862dbc1 P 78b9263889cd4f1582578a3aa4f644d9: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-1-root/wals/0be6c610fe68472b8bcb6232b862dbc1
I20260808 20:13:35.940758 13079 raft_consensus.cc:2932] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.947575 13078 raft_consensus.cc:2932] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.952373 13236 raft_consensus.cc:2229] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:35.954942 11894 catalog_manager.cc:5495] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.963165 12244 raft_consensus.cc:2932] T f4918de0ffe94198a66ac1975ddbc693 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.964193 13236 raft_consensus.cc:2258] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:35.967013 13236 ts_tablet_manager.cc:1905] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 4cd46f13bceb4c3c9ef71e755e17b995: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:35.975517 11875 catalog_manager.cc:4841] TS 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): tablet 0be6c610fe68472b8bcb6232b862dbc1 (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:35.990399 11892 catalog_manager.cc:5495] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:35.994165 13236 ts_tablet_manager.cc:1918] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 4cd46f13bceb4c3c9ef71e755e17b995: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:35.994608 13236 log.cc:1196] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 4cd46f13bceb4c3c9ef71e755e17b995: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-0-root/wals/1d7d7a85e3964f3f9d5f2dc39e23340b
I20260808 20:13:35.995972 11873 catalog_manager.cc:4841] TS 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): tablet 1d7d7a85e3964f3f9d5f2dc39e23340b (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:36.021615 12152 consensus_queue.cc:237] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:36.026944 12147 consensus_queue.cc:237] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:36.032286 12244 raft_consensus.cc:1261] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:36.032385 12043 raft_consensus.cc:1261] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:36.032438 13078 raft_consensus.cc:1261] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:36.032485 13079 raft_consensus.cc:1261] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:36.033687 13157 consensus_queue.cc:1035] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:36.034133 13211 consensus_queue.cc:1035] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 20:13:36.036640 13225 consensus_queue.cc:1035] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:36.043587 13225 raft_consensus.cc:2932] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:36.045248 13078 raft_consensus.cc:2932] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:36.047102 12038 raft_consensus.cc:2932] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:36.050196 13158 raft_consensus.cc:2932] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:36.056030 12245 raft_consensus.cc:2932] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:36.059448 11875 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e256c29dec964f49856cf204f950793b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:36.061072 13211 consensus_queue.cc:1035] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 20:13:36.066957 13079 raft_consensus.cc:2932] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:36.067806 11875 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f4918de0ffe94198a66ac1975ddbc693 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:36.068403 11892 catalog_manager.cc:5495] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:36.070133 11896 catalog_manager.cc:5495] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:36.082695 12235 tablet_service.cc:1514] Processing DeleteTablet for tablet f4918de0ffe94198a66ac1975ddbc693 with delete_type TABLET_DATA_TOMBSTONED (TS b9d94f35ea15421aa1d721201c782d8a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44376
I20260808 20:13:36.085309 13240 tablet_replica.cc:330] T f4918de0ffe94198a66ac1975ddbc693 P b9d94f35ea15421aa1d721201c782d8a: stopping tablet replica
I20260808 20:13:36.086323 13240 raft_consensus.cc:2229] T f4918de0ffe94198a66ac1975ddbc693 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:36.087469 13240 raft_consensus.cc:2258] T f4918de0ffe94198a66ac1975ddbc693 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:36.088011 12022 tablet_service.cc:1514] Processing DeleteTablet for tablet e256c29dec964f49856cf204f950793b with delete_type TABLET_DATA_TOMBSTONED (TS 4cd46f13bceb4c3c9ef71e755e17b995 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53014
I20260808 20:13:36.088971 13236 tablet_replica.cc:330] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995: stopping tablet replica
I20260808 20:13:36.089970 13236 raft_consensus.cc:2229] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:36.090605 13236 raft_consensus.cc:2258] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:36.093843 13236 ts_tablet_manager.cc:1905] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:36.100728 13240 ts_tablet_manager.cc:1905] T f4918de0ffe94198a66ac1975ddbc693 P b9d94f35ea15421aa1d721201c782d8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:36.116595 13236 ts_tablet_manager.cc:1918] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:36.117085 13236 log.cc:1196] T e256c29dec964f49856cf204f950793b P 4cd46f13bceb4c3c9ef71e755e17b995: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-0-root/wals/e256c29dec964f49856cf204f950793b
I20260808 20:13:36.118505 11873 catalog_manager.cc:4841] TS 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): tablet e256c29dec964f49856cf204f950793b (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:36.127142 13240 ts_tablet_manager.cc:1918] T f4918de0ffe94198a66ac1975ddbc693 P b9d94f35ea15421aa1d721201c782d8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:36.127581 13240 log.cc:1196] T f4918de0ffe94198a66ac1975ddbc693 P b9d94f35ea15421aa1d721201c782d8a: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-2-root/wals/f4918de0ffe94198a66ac1975ddbc693
I20260808 20:13:36.129202 11872 catalog_manager.cc:4841] TS b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): tablet f4918de0ffe94198a66ac1975ddbc693 (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
W20260808 20:13:37.275476 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 50369us
I20260808 20:13:37.329105 12045 consensus_queue.cc:237] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:37.334050 12244 raft_consensus.cc:1261] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:37.334311 12146 raft_consensus.cc:1261] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 20:13:37.334556 12001 consensus_peers.cc:487] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 940aa50483d844deb4031f312b4a7355. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:37.335163 13186 consensus_queue.cc:1035] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:37.335857 13242 consensus_queue.cc:1035] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:37.343488 13245 raft_consensus.cc:2932] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:37.345456 12146 raft_consensus.cc:2932] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:37.346283 12244 raft_consensus.cc:2932] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:37.355981 12045 consensus_queue.cc:237] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:37.362406 11894 catalog_manager.cc:5495] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260808 20:13:37.372442 12001 consensus_peers.cc:487] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 102eea36004446e5a71aaccebb9ad859. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:37.372613 12248 raft_consensus.cc:1261] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:37.372717 12152 raft_consensus.cc:1261] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:37.374047 13189 consensus_queue.cc:1035] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:37.375991 13245 consensus_queue.cc:1035] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 20:13:37.393463 13189 raft_consensus.cc:2932] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:37.394946 12248 raft_consensus.cc:2932] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:37.410276 12146 raft_consensus.cc:2932] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:37.412694 12152 consensus_queue.cc:237] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:37.421541 12248 raft_consensus.cc:1261] T a383bcab83014009b5698c63cdac298b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:37.422674 12045 raft_consensus.cc:1261] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:37.422811 13253 consensus_queue.cc:1035] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:37.424127 13157 consensus_queue.cc:1035] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260808 20:13:37.426498 12105 consensus_peers.cc:487] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a383bcab83014009b5698c63cdac298b. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:37.427098 11897 catalog_manager.cc:5495] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:37.436718 13259 raft_consensus.cc:2932] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:37.440347 12038 raft_consensus.cc:2932] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:37.450448 12045 consensus_queue.cc:237] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:37.458235 11894 catalog_manager.cc:5495] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:37.458911 12248 raft_consensus.cc:2932] T a383bcab83014009b5698c63cdac298b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:37.467808 12244 raft_consensus.cc:1261] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:37.470269 12152 raft_consensus.cc:1261] T 1e0d98ab3b4945d8b845711f4ddd7153 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:37.471859 13189 consensus_queue.cc:1035] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:37.472740 13243 consensus_queue.cc:1035] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260808 20:13:37.474179 12001 consensus_peers.cc:487] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e0d98ab3b4945d8b845711f4ddd7153. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:37.488534 13243 raft_consensus.cc:2932] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:37.489686 12152 raft_consensus.cc:2932] T 1e0d98ab3b4945d8b845711f4ddd7153 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:37.496222 12248 raft_consensus.cc:2932] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:37.506551 11892 catalog_manager.cc:5495] T 1e0d98ab3b4945d8b845711f4ddd7153 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:37.794728 13284 ts_tablet_manager.cc:927] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:13:37.796028 13284 tablet_copy_client.cc:316] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.1:40747
I20260808 20:13:37.797256 12062 tablet_copy_service.cc:140] P 4cd46f13bceb4c3c9ef71e755e17b995: Received BeginTabletCopySession request for tablet 102eea36004446e5a71aaccebb9ad859 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:49226)
I20260808 20:13:37.797658 12062 tablet_copy_service.cc:161] P 4cd46f13bceb4c3c9ef71e755e17b995: Beginning new tablet copy session on tablet 102eea36004446e5a71aaccebb9ad859 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:49226: session id = e5bd9a4616ce4acfa20dc7a002984a20-102eea36004446e5a71aaccebb9ad859
I20260808 20:13:37.804505 12062 tablet_copy_source_session.cc:205] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:37.807448 13284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 102eea36004446e5a71aaccebb9ad859. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:37.818905 13284 tablet_copy_client.cc:791] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:37.819387 13284 tablet_copy_client.cc:655] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:37.822961 13284 tablet_copy_client.cc:531] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:37.831004 13284 tablet_bootstrap.cc:492] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:37.849033 13284 tablet_bootstrap.cc:492] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:37.849895 13284 tablet_bootstrap.cc:492] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:37.850589 13284 ts_tablet_manager.cc:1397] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.000s
I20260808 20:13:37.852960 13284 raft_consensus.cc:348] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:37.853767 13284 raft_consensus.cc:729] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:37.854360 13284 consensus_queue.cc:260] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:37.857013 13284 ts_tablet_manager.cc:1428] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 20:13:37.858667 12062 tablet_copy_service.cc:342] P 4cd46f13bceb4c3c9ef71e755e17b995: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-102eea36004446e5a71aaccebb9ad859 received from {username='slave'} at 127.0.0.1:49226
I20260808 20:13:37.859022 12062 tablet_copy_service.cc:433] P 4cd46f13bceb4c3c9ef71e755e17b995: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-102eea36004446e5a71aaccebb9ad859 on tablet 102eea36004446e5a71aaccebb9ad859 with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:37.877467 13284 ts_tablet_manager.cc:927] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:13:37.878894 13284 tablet_copy_client.cc:316] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.1:40747
I20260808 20:13:37.880071 12062 tablet_copy_service.cc:140] P 4cd46f13bceb4c3c9ef71e755e17b995: Received BeginTabletCopySession request for tablet 940aa50483d844deb4031f312b4a7355 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:49226)
I20260808 20:13:37.880494 12062 tablet_copy_service.cc:161] P 4cd46f13bceb4c3c9ef71e755e17b995: Beginning new tablet copy session on tablet 940aa50483d844deb4031f312b4a7355 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:49226: session id = e5bd9a4616ce4acfa20dc7a002984a20-940aa50483d844deb4031f312b4a7355
I20260808 20:13:37.886250 12062 tablet_copy_source_session.cc:205] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:37.889185 13284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 940aa50483d844deb4031f312b4a7355. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:37.902325 13284 tablet_copy_client.cc:791] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:37.902805 13284 tablet_copy_client.cc:655] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:37.905813 13284 tablet_copy_client.cc:531] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:37.912387 13293 ts_tablet_manager.cc:927] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:37.914533 13284 tablet_bootstrap.cc:492] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:37.925909 13293 tablet_copy_client.cc:316] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.2:35139
I20260808 20:13:37.927415 12164 tablet_copy_service.cc:140] P 78b9263889cd4f1582578a3aa4f644d9: Received BeginTabletCopySession request for tablet a383bcab83014009b5698c63cdac298b from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:46680)
I20260808 20:13:37.927904 12164 tablet_copy_service.cc:161] P 78b9263889cd4f1582578a3aa4f644d9: Beginning new tablet copy session on tablet a383bcab83014009b5698c63cdac298b from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:46680: session id = e5bd9a4616ce4acfa20dc7a002984a20-a383bcab83014009b5698c63cdac298b
I20260808 20:13:37.935215 13284 tablet_bootstrap.cc:492] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:37.936123 13284 tablet_bootstrap.cc:492] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:37.936208 12164 tablet_copy_source_session.cc:205] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:37.936800 13284 ts_tablet_manager.cc:1397] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.004s
I20260808 20:13:37.939221 13293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a383bcab83014009b5698c63cdac298b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:37.939482 13284 raft_consensus.cc:348] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:37.940335 13284 raft_consensus.cc:729] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:37.940991 13284 consensus_queue.cc:260] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:37.944938 13284 ts_tablet_manager.cc:1428] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260808 20:13:37.946297 12062 tablet_copy_service.cc:342] P 4cd46f13bceb4c3c9ef71e755e17b995: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-940aa50483d844deb4031f312b4a7355 received from {username='slave'} at 127.0.0.1:49226
I20260808 20:13:37.946590 12062 tablet_copy_service.cc:433] P 4cd46f13bceb4c3c9ef71e755e17b995: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-940aa50483d844deb4031f312b4a7355 on tablet 940aa50483d844deb4031f312b4a7355 with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:37.954813 13293 tablet_copy_client.cc:791] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:37.955330 13293 tablet_copy_client.cc:655] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:37.958258 13293 tablet_copy_client.cc:531] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:37.964787 13293 tablet_bootstrap.cc:492] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
W20260808 20:13:37.974815 12182 debug-util.cc:398] Leaking SignalData structure 0x7b08000f0a80 after lost signal to thread 10340
W20260808 20:13:37.975804 12182 debug-util.cc:398] Leaking SignalData structure 0x7b080016d780 after lost signal to thread 11946
W20260808 20:13:37.976922 12182 debug-util.cc:398] Leaking SignalData structure 0x7b08002fc000 after lost signal to thread 12083
W20260808 20:13:37.977872 12182 debug-util.cc:398] Leaking SignalData structure 0x7b080027bd60 after lost signal to thread 12185
W20260808 20:13:37.978835 12182 debug-util.cc:398] Leaking SignalData structure 0x7b0800183060 after lost signal to thread 12287
W20260808 20:13:37.979795 12182 debug-util.cc:398] Leaking SignalData structure 0x7b08000aab80 after lost signal to thread 13118
I20260808 20:13:38.000754 13293 tablet_bootstrap.cc:492] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:38.001780 13293 tablet_bootstrap.cc:492] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:38.002194 13284 ts_tablet_manager.cc:927] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:13:38.002595 13293 ts_tablet_manager.cc:1397] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.038s user 0.022s sys 0.017s
I20260808 20:13:38.005415 13293 raft_consensus.cc:348] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:38.006385 13293 raft_consensus.cc:729] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:38.007122 13293 consensus_queue.cc:260] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:38.011768 13293 ts_tablet_manager.cc:1428] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260808 20:13:38.013923 12164 tablet_copy_service.cc:342] P 78b9263889cd4f1582578a3aa4f644d9: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-a383bcab83014009b5698c63cdac298b received from {username='slave'} at 127.0.0.1:46680
I20260808 20:13:38.014317 12164 tablet_copy_service.cc:433] P 78b9263889cd4f1582578a3aa4f644d9: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-a383bcab83014009b5698c63cdac298b on tablet a383bcab83014009b5698c63cdac298b with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:38.015626 13284 tablet_copy_client.cc:316] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.1:40747
I20260808 20:13:38.018962 12062 tablet_copy_service.cc:140] P 4cd46f13bceb4c3c9ef71e755e17b995: Received BeginTabletCopySession request for tablet 1e0d98ab3b4945d8b845711f4ddd7153 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:49226)
I20260808 20:13:38.019367 12062 tablet_copy_service.cc:161] P 4cd46f13bceb4c3c9ef71e755e17b995: Beginning new tablet copy session on tablet 1e0d98ab3b4945d8b845711f4ddd7153 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:49226: session id = e5bd9a4616ce4acfa20dc7a002984a20-1e0d98ab3b4945d8b845711f4ddd7153
I20260808 20:13:38.030335 12062 tablet_copy_source_session.cc:205] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:38.034621 13284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e0d98ab3b4945d8b845711f4ddd7153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:38.049750 13284 tablet_copy_client.cc:791] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:38.050417 13284 tablet_copy_client.cc:655] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:38.053855 13284 tablet_copy_client.cc:531] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:38.068722 13284 tablet_bootstrap.cc:492] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:38.095566 13284 tablet_bootstrap.cc:492] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:38.096534 13284 tablet_bootstrap.cc:492] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:38.097220 13284 ts_tablet_manager.cc:1397] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.029s user 0.017s sys 0.007s
I20260808 20:13:38.100029 13284 raft_consensus.cc:348] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:38.100816 13284 raft_consensus.cc:729] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:38.101464 13284 consensus_queue.cc:260] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:38.104146 13284 ts_tablet_manager.cc:1428] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260808 20:13:38.107951 12062 tablet_copy_service.cc:342] P 4cd46f13bceb4c3c9ef71e755e17b995: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-1e0d98ab3b4945d8b845711f4ddd7153 received from {username='slave'} at 127.0.0.1:49226
I20260808 20:13:38.108318 12062 tablet_copy_service.cc:433] P 4cd46f13bceb4c3c9ef71e755e17b995: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-1e0d98ab3b4945d8b845711f4ddd7153 on tablet 1e0d98ab3b4945d8b845711f4ddd7153 with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:38.343140 13078 raft_consensus.cc:1203] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:38.413745 13078 raft_consensus.cc:1203] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:38.434562 13078 raft_consensus.cc:1203] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:38.544013 13078 raft_consensus.cc:1203] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:38.895911 13302 raft_consensus.cc:1050] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:38.899349 13302 consensus_queue.cc:237] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:38.930804 13078 raft_consensus.cc:1261] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:38.933148 13189 raft_consensus.cc:1050] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:38.937111 13312 consensus_queue.cc:1035] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:13:38.939721 12248 raft_consensus.cc:1261] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:38.942778 12147 raft_consensus.cc:1261] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:38.946113 13189 consensus_queue.cc:237] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:38.951381 13312 raft_consensus.cc:1050] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:38.949229 13311 consensus_queue.cc:1035] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260808 20:13:38.953912 13242 consensus_queue.cc:1035] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:38.955329 13312 consensus_queue.cc:237] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:38.989877 13078 raft_consensus.cc:1261] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:38.992067 12245 raft_consensus.cc:1261] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:38.995599 13319 consensus_queue.cc:1035] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:38.997932 13319 consensus_queue.cc:1035] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:39.019038 13080 raft_consensus.cc:1261] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:39.020169 13315 raft_consensus.cc:2932] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.024046 12248 raft_consensus.cc:1261] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:39.025918 13312 consensus_queue.cc:1035] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:39.026928 13189 consensus_queue.cc:1035] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:39.026783 13080 raft_consensus.cc:2932] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
W20260808 20:13:39.037902 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 54400.9us
I20260808 20:13:39.040001 12147 raft_consensus.cc:1261] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:39.043660 12147 raft_consensus.cc:1261] T 1e0d98ab3b4945d8b845711f4ddd7153 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:39.062578 13310 consensus_queue.cc:1035] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:13:39.055184 13189 consensus_queue.cc:1035] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20260808 20:13:39.090505 12250 raft_consensus.cc:2932] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.105805 12146 raft_consensus.cc:2932] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
W20260808 20:13:39.134464 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 80008.8us
I20260808 20:13:39.134797 11892 catalog_manager.cc:5495] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260808 20:13:39.161922 13330 log.cc:924] Time spent T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20260808 20:13:39.186282 13253 raft_consensus.cc:1050] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:39.188778 13253 consensus_queue.cc:237] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:39.215032 13310 raft_consensus.cc:2932] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.216698 12146 raft_consensus.cc:2932] T 1e0d98ab3b4945d8b845711f4ddd7153 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
W20260808 20:13:39.217720 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 81989.6us
I20260808 20:13:39.221961 12152 raft_consensus.cc:2932] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.219389 13315 raft_consensus.cc:2932] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.255298 12038 consensus_queue.cc:237] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:39.262131 12245 raft_consensus.cc:2932] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.267107 12249 raft_consensus.cc:2932] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.275513 11889 catalog_manager.cc:5495] T 1e0d98ab3b4945d8b845711f4ddd7153 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.276503 13080 raft_consensus.cc:2932] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.285207 12248 raft_consensus.cc:1261] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:39.286412 13315 consensus_queue.cc:1035] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:39.281461 13079 raft_consensus.cc:1261] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:39.292284 13320 consensus_queue.cc:1035] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:39.292116 13189 raft_consensus.cc:2932] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.292625 11889 catalog_manager.cc:5495] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.296928 13079 raft_consensus.cc:2932] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.302268 12250 raft_consensus.cc:2932] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.305686 11873 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 940aa50483d844deb4031f312b4a7355 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:39.308740 13080 raft_consensus.cc:1261] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:39.310096 12038 raft_consensus.cc:1261] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:39.311123 13253 consensus_queue.cc:1035] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:39.315454 12248 raft_consensus.cc:1261] T a383bcab83014009b5698c63cdac298b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:39.317214 13253 consensus_queue.cc:1035] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:39.321139 13253 consensus_queue.cc:1035] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:39.330331 11894 catalog_manager.cc:5495] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.321843 13080 raft_consensus.cc:2932] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.344395 13253 raft_consensus.cc:2932] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.345372 12250 raft_consensus.cc:2932] T a383bcab83014009b5698c63cdac298b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.369122 12131 tablet_service.cc:1514] Processing DeleteTablet for tablet 940aa50483d844deb4031f312b4a7355 with delete_type TABLET_DATA_TOMBSTONED (TS 78b9263889cd4f1582578a3aa4f644d9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53148
I20260808 20:13:39.366745 13080 raft_consensus.cc:2932] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.368947 11897 catalog_manager.cc:5495] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:39.376824 13365 tablet_replica.cc:330] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9: stopping tablet replica
I20260808 20:13:39.379403 13365 raft_consensus.cc:2229] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:39.384487 13365 raft_consensus.cc:2258] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:39.389088 12038 consensus_queue.cc:237] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:39.394546 13365 ts_tablet_manager.cc:1905] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:39.395412 13078 raft_consensus.cc:1261] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:39.394745 12038 raft_consensus.cc:2932] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.397924 12152 raft_consensus.cc:1261] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:39.398892 13310 consensus_queue.cc:1035] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:39.401407 13189 consensus_queue.cc:1035] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:39.404634 12046 consensus_queue.cc:237] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:39.423190 13310 raft_consensus.cc:2932] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.426951 12248 raft_consensus.cc:1261] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:39.427510 13365 ts_tablet_manager.cc:1918] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:39.427961 13365 log.cc:1196] T 940aa50483d844deb4031f312b4a7355 P 78b9263889cd4f1582578a3aa4f644d9: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-1-root/wals/940aa50483d844deb4031f312b4a7355
I20260808 20:13:39.428352 13315 consensus_queue.cc:1035] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:39.431010 11875 catalog_manager.cc:4841] TS 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): tablet 940aa50483d844deb4031f312b4a7355 (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:39.442457 12146 raft_consensus.cc:2932] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.462008 13078 raft_consensus.cc:1261] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:39.463306 13320 consensus_queue.cc:1035] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:39.468108 13080 raft_consensus.cc:2932] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.470314 11873 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 102eea36004446e5a71aaccebb9ad859 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:39.470559 13189 raft_consensus.cc:2932] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.480655 12152 consensus_queue.cc:237] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:39.493825 11873 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 1e0d98ab3b4945d8b845711f4ddd7153 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:39.494333 12250 raft_consensus.cc:2932] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.518653 12038 raft_consensus.cc:1261] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:39.521552 13345 consensus_queue.cc:1035] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:39.501791 13078 raft_consensus.cc:2932] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.537348 13304 raft_consensus.cc:2932] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.540944 12046 raft_consensus.cc:2932] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.545485 13080 raft_consensus.cc:1261] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:39.546734 13345 consensus_queue.cc:1035] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:39.549511 11875 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet a383bcab83014009b5698c63cdac298b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:39.553133 13080 raft_consensus.cc:2932] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.557279 11897 catalog_manager.cc:5495] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:39.555845 11889 catalog_manager.cc:5495] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.575338 12235 tablet_service.cc:1514] Processing DeleteTablet for tablet 102eea36004446e5a71aaccebb9ad859 with delete_type TABLET_DATA_TOMBSTONED (TS b9d94f35ea15421aa1d721201c782d8a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44376
I20260808 20:13:39.587041 13369 tablet_replica.cc:330] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a: stopping tablet replica
I20260808 20:13:39.588176 13369 raft_consensus.cc:2229] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:39.588882 13369 raft_consensus.cc:2258] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:39.594702 11897 catalog_manager.cc:5495] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:39.597980 13369 ts_tablet_manager.cc:1905] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:39.606990 12131 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e0d98ab3b4945d8b845711f4ddd7153 with delete_type TABLET_DATA_TOMBSTONED (TS 78b9263889cd4f1582578a3aa4f644d9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53148
I20260808 20:13:39.608724 13365 tablet_replica.cc:330] T 1e0d98ab3b4945d8b845711f4ddd7153 P 78b9263889cd4f1582578a3aa4f644d9: stopping tablet replica
I20260808 20:13:39.609694 13365 raft_consensus.cc:2229] T 1e0d98ab3b4945d8b845711f4ddd7153 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:39.610379 13365 raft_consensus.cc:2258] T 1e0d98ab3b4945d8b845711f4ddd7153 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:39.612488 13369 ts_tablet_manager.cc:1918] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:39.613008 13369 log.cc:1196] T 102eea36004446e5a71aaccebb9ad859 P b9d94f35ea15421aa1d721201c782d8a: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-2-root/wals/102eea36004446e5a71aaccebb9ad859
I20260808 20:13:39.614073 13365 ts_tablet_manager.cc:1905] T 1e0d98ab3b4945d8b845711f4ddd7153 P 78b9263889cd4f1582578a3aa4f644d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:39.615375 11872 catalog_manager.cc:4841] TS b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): tablet 102eea36004446e5a71aaccebb9ad859 (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:39.616407 12235 tablet_service.cc:1514] Processing DeleteTablet for tablet a383bcab83014009b5698c63cdac298b with delete_type TABLET_DATA_TOMBSTONED (TS b9d94f35ea15421aa1d721201c782d8a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44376
I20260808 20:13:39.617264 13369 tablet_replica.cc:330] T a383bcab83014009b5698c63cdac298b P b9d94f35ea15421aa1d721201c782d8a: stopping tablet replica
I20260808 20:13:39.617997 13369 raft_consensus.cc:2229] T a383bcab83014009b5698c63cdac298b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:39.618563 13369 raft_consensus.cc:2258] T a383bcab83014009b5698c63cdac298b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:39.621206 13369 ts_tablet_manager.cc:1905] T a383bcab83014009b5698c63cdac298b P b9d94f35ea15421aa1d721201c782d8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:39.629710 13365 ts_tablet_manager.cc:1918] T 1e0d98ab3b4945d8b845711f4ddd7153 P 78b9263889cd4f1582578a3aa4f644d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:39.630582 13365 log.cc:1196] T 1e0d98ab3b4945d8b845711f4ddd7153 P 78b9263889cd4f1582578a3aa4f644d9: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-1-root/wals/1e0d98ab3b4945d8b845711f4ddd7153
I20260808 20:13:39.632211 11875 catalog_manager.cc:4841] TS 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): tablet 1e0d98ab3b4945d8b845711f4ddd7153 (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:39.635505 13369 ts_tablet_manager.cc:1918] T a383bcab83014009b5698c63cdac298b P b9d94f35ea15421aa1d721201c782d8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:39.635967 13369 log.cc:1196] T a383bcab83014009b5698c63cdac298b P b9d94f35ea15421aa1d721201c782d8a: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-2-root/wals/a383bcab83014009b5698c63cdac298b
I20260808 20:13:39.637140 11872 catalog_manager.cc:4841] TS b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): tablet a383bcab83014009b5698c63cdac298b (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:40.645540 12248 consensus_queue.cc:237] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:40.651324 12152 raft_consensus.cc:1261] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:40.654006 13247 consensus_queue.cc:1035] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:40.657240 12038 raft_consensus.cc:1261] T 4e393e90875847809e1ef94179c5b4ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:40.658504 13247 consensus_queue.cc:1035] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:40.667811 12152 raft_consensus.cc:2932] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:40.666877 13247 raft_consensus.cc:2932] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:40.692692 12146 consensus_queue.cc:237] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
W20260808 20:13:40.702700 12207 consensus_peers.cc:487] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e393e90875847809e1ef94179c5b4ef. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:40.704173 11897 catalog_manager.cc:5495] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:40.725486 12248 raft_consensus.cc:1261] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:40.726913 13304 consensus_queue.cc:1035] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 20:13:40.732136 12038 raft_consensus.cc:1261] T f73cf4b21a754e75b2af936d7402def2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:40.734400 12038 raft_consensus.cc:2932] T 4e393e90875847809e1ef94179c5b4ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:40.733412 13389 consensus_queue.cc:1035] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
W20260808 20:13:40.742815 12105 consensus_peers.cc:487] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f73cf4b21a754e75b2af936d7402def2. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:40.763190 13389 raft_consensus.cc:2932] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:40.769670 12248 raft_consensus.cc:2932] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:40.778611 12046 raft_consensus.cc:2932] T f73cf4b21a754e75b2af936d7402def2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:40.783502 12250 consensus_queue.cc:237] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:40.833150 11894 catalog_manager.cc:5495] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:40.844518 12152 raft_consensus.cc:1261] T 72e59eff611e4cc693407979ebe0857c P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 20:13:40.845510 12207 consensus_peers.cc:487] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72e59eff611e4cc693407979ebe0857c. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:40.846756 13247 consensus_queue.cc:1035] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:40.850162 12046 raft_consensus.cc:1261] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:40.854079 13247 consensus_queue.cc:1035] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:40.864364 13380 raft_consensus.cc:2932] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:40.870734 12146 raft_consensus.cc:2932] T 72e59eff611e4cc693407979ebe0857c P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:40.880213 12039 raft_consensus.cc:2932] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:40.890128 11896 catalog_manager.cc:5495] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:40.897292 12152 consensus_queue.cc:237] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:40.906535 12250 raft_consensus.cc:1261] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:40.907820 13304 consensus_queue.cc:1035] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260808 20:13:40.911415 12105 consensus_peers.cc:487] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd0cef71281c44f6b3fd2afe876735f8. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:40.912096 12039 raft_consensus.cc:1261] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:40.913116 13386 consensus_queue.cc:1035] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:40.935985 13389 raft_consensus.cc:2932] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:40.938606 12039 raft_consensus.cc:2932] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:40.938700 12250 raft_consensus.cc:2932] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:40.956440 11894 catalog_manager.cc:5495] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:41.159924 13414 ts_tablet_manager.cc:927] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:41.161566 13414 tablet_copy_client.cc:316] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.3:38391
I20260808 20:13:41.203270 12266 tablet_copy_service.cc:140] P b9d94f35ea15421aa1d721201c782d8a: Received BeginTabletCopySession request for tablet 4e393e90875847809e1ef94179c5b4ef from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:53030)
I20260808 20:13:41.203804 12266 tablet_copy_service.cc:161] P b9d94f35ea15421aa1d721201c782d8a: Beginning new tablet copy session on tablet 4e393e90875847809e1ef94179c5b4ef from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:53030: session id = e5bd9a4616ce4acfa20dc7a002984a20-4e393e90875847809e1ef94179c5b4ef
I20260808 20:13:41.209684 12266 tablet_copy_source_session.cc:205] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:41.212543 13414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e393e90875847809e1ef94179c5b4ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:41.228021 13414 tablet_copy_client.cc:791] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:41.228716 13414 tablet_copy_client.cc:655] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:41.231940 13414 tablet_copy_client.cc:531] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:41.241153 13414 tablet_bootstrap.cc:492] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:41.306982 13414 tablet_bootstrap.cc:492] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:41.308043 13414 tablet_bootstrap.cc:492] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:41.309240 13414 ts_tablet_manager.cc:1397] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.068s user 0.005s sys 0.023s
I20260808 20:13:41.312058 13414 raft_consensus.cc:348] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:41.313372 13414 raft_consensus.cc:729] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:41.314069 13414 consensus_queue.cc:260] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:41.342597 13414 ts_tablet_manager.cc:1428] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.033s user 0.006s sys 0.006s
I20260808 20:13:41.351683 13426 ts_tablet_manager.cc:927] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:41.369091 13428 ts_tablet_manager.cc:927] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:41.369124 12266 tablet_copy_service.cc:342] P b9d94f35ea15421aa1d721201c782d8a: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-4e393e90875847809e1ef94179c5b4ef received from {username='slave'} at 127.0.0.1:53030
I20260808 20:13:41.370632 12266 tablet_copy_service.cc:433] P b9d94f35ea15421aa1d721201c782d8a: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-4e393e90875847809e1ef94179c5b4ef on tablet 4e393e90875847809e1ef94179c5b4ef with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:41.430384 13426 tablet_copy_client.cc:316] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.3:38391
I20260808 20:13:41.431857 12266 tablet_copy_service.cc:140] P b9d94f35ea15421aa1d721201c782d8a: Received BeginTabletCopySession request for tablet 72e59eff611e4cc693407979ebe0857c from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:53030)
I20260808 20:13:41.432276 12266 tablet_copy_service.cc:161] P b9d94f35ea15421aa1d721201c782d8a: Beginning new tablet copy session on tablet 72e59eff611e4cc693407979ebe0857c from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:53030: session id = e5bd9a4616ce4acfa20dc7a002984a20-72e59eff611e4cc693407979ebe0857c
I20260808 20:13:41.440171 12266 tablet_copy_source_session.cc:205] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:41.455369 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e59eff611e4cc693407979ebe0857c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:41.479754 13428 tablet_copy_client.cc:316] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.2:35139
I20260808 20:13:41.481184 12164 tablet_copy_service.cc:140] P 78b9263889cd4f1582578a3aa4f644d9: Received BeginTabletCopySession request for tablet f73cf4b21a754e75b2af936d7402def2 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:46680)
I20260808 20:13:41.481590 12164 tablet_copy_service.cc:161] P 78b9263889cd4f1582578a3aa4f644d9: Beginning new tablet copy session on tablet f73cf4b21a754e75b2af936d7402def2 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:46680: session id = e5bd9a4616ce4acfa20dc7a002984a20-f73cf4b21a754e75b2af936d7402def2
I20260808 20:13:41.487876 12164 tablet_copy_source_session.cc:205] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:41.490875 13428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f73cf4b21a754e75b2af936d7402def2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:41.494644 13426 tablet_copy_client.cc:791] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:41.496908 13426 tablet_copy_client.cc:655] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:41.505591 13414 ts_tablet_manager.cc:927] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:41.507529 13428 tablet_copy_client.cc:791] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:41.508066 13428 tablet_copy_client.cc:655] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:41.508531 13426 tablet_copy_client.cc:531] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:41.514484 13414 tablet_copy_client.cc:316] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.2:35139
I20260808 20:13:41.516765 12163 tablet_copy_service.cc:140] P 78b9263889cd4f1582578a3aa4f644d9: Received BeginTabletCopySession request for tablet fd0cef71281c44f6b3fd2afe876735f8 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:46680)
I20260808 20:13:41.517295 12163 tablet_copy_service.cc:161] P 78b9263889cd4f1582578a3aa4f644d9: Beginning new tablet copy session on tablet fd0cef71281c44f6b3fd2afe876735f8 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:46680: session id = e5bd9a4616ce4acfa20dc7a002984a20-fd0cef71281c44f6b3fd2afe876735f8
I20260808 20:13:41.517987 13428 tablet_copy_client.cc:531] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:41.525234 12163 tablet_copy_source_session.cc:205] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:41.516571 13426 tablet_bootstrap.cc:492] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:41.527099 13428 tablet_bootstrap.cc:492] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:41.527935 13414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0cef71281c44f6b3fd2afe876735f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:41.552862 13426 tablet_bootstrap.cc:492] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:41.553799 13426 tablet_bootstrap.cc:492] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:41.554517 13426 ts_tablet_manager.cc:1397] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.038s user 0.018s sys 0.007s
I20260808 20:13:41.557145 13426 raft_consensus.cc:348] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:41.558053 13426 raft_consensus.cc:729] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:41.558701 13426 consensus_queue.cc:260] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:41.566913 13428 tablet_bootstrap.cc:492] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:41.567449 13426 ts_tablet_manager.cc:1428] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260808 20:13:41.567852 13428 tablet_bootstrap.cc:492] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:41.568675 13428 ts_tablet_manager.cc:1397] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.042s user 0.017s sys 0.014s
I20260808 20:13:41.571230 13428 raft_consensus.cc:348] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:41.572098 13428 raft_consensus.cc:729] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:41.572747 13428 consensus_queue.cc:260] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:41.573426 12266 tablet_copy_service.cc:342] P b9d94f35ea15421aa1d721201c782d8a: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-72e59eff611e4cc693407979ebe0857c received from {username='slave'} at 127.0.0.1:53030
I20260808 20:13:41.573827 12266 tablet_copy_service.cc:433] P b9d94f35ea15421aa1d721201c782d8a: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-72e59eff611e4cc693407979ebe0857c on tablet 72e59eff611e4cc693407979ebe0857c with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:41.576292 13428 ts_tablet_manager.cc:1428] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260808 20:13:41.580243 12163 tablet_copy_service.cc:342] P 78b9263889cd4f1582578a3aa4f644d9: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-f73cf4b21a754e75b2af936d7402def2 received from {username='slave'} at 127.0.0.1:46680
I20260808 20:13:41.580605 12163 tablet_copy_service.cc:433] P 78b9263889cd4f1582578a3aa4f644d9: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-f73cf4b21a754e75b2af936d7402def2 on tablet f73cf4b21a754e75b2af936d7402def2 with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:41.589779 13414 tablet_copy_client.cc:791] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:41.590406 13414 tablet_copy_client.cc:655] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:41.597818 13414 tablet_copy_client.cc:531] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:41.607354 13414 tablet_bootstrap.cc:492] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:41.635243 13414 tablet_bootstrap.cc:492] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:41.636178 13414 tablet_bootstrap.cc:492] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:41.636878 13414 ts_tablet_manager.cc:1397] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.013s
I20260808 20:13:41.639756 13414 raft_consensus.cc:348] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:41.640661 13414 raft_consensus.cc:729] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:41.641373 13414 consensus_queue.cc:260] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:41.648307 13414 ts_tablet_manager.cc:1428] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260808 20:13:41.649904 12163 tablet_copy_service.cc:342] P 78b9263889cd4f1582578a3aa4f644d9: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-fd0cef71281c44f6b3fd2afe876735f8 received from {username='slave'} at 127.0.0.1:46680
I20260808 20:13:41.650316 12163 tablet_copy_service.cc:433] P 78b9263889cd4f1582578a3aa4f644d9: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-fd0cef71281c44f6b3fd2afe876735f8 on tablet fd0cef71281c44f6b3fd2afe876735f8 with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:41.729910 13080 raft_consensus.cc:1203] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:41.882640 13080 raft_consensus.cc:1203] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:41.920449 13080 raft_consensus.cc:1203] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:41.942037 13080 raft_consensus.cc:1203] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:42.203214 13444 raft_consensus.cc:1050] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:42.208519 13444 consensus_queue.cc:237] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:42.348752 12146 raft_consensus.cc:1261] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:42.350383 13080 raft_consensus.cc:1261] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:42.353848 13382 consensus_queue.cc:1035] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:13:42.356024 13380 consensus_queue.cc:1035] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:13:42.358979 13444 raft_consensus.cc:1050] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:42.363190 13444 consensus_queue.cc:237] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:42.366000 13389 raft_consensus.cc:1050] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:42.372344 13450 raft_consensus.cc:1050] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:42.372917 13389 consensus_queue.cc:237] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:42.375738 13450 consensus_queue.cc:237] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:42.378362 12046 raft_consensus.cc:1261] T 4e393e90875847809e1ef94179c5b4ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:42.379997 13380 consensus_queue.cc:1035] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:42.394285 13078 raft_consensus.cc:1261] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:42.401011 12146 raft_consensus.cc:1261] T 72e59eff611e4cc693407979ebe0857c P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:42.402659 13247 consensus_queue.cc:1035] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:13:42.405519 12046 raft_consensus.cc:1261] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:42.428422 13382 consensus_queue.cc:1035] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:42.436924 12248 raft_consensus.cc:1261] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:42.438670 13439 consensus_queue.cc:1035] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:13:42.439282 13079 raft_consensus.cc:1261] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:42.449470 13439 consensus_queue.cc:1035] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260808 20:13:42.481470 12045 raft_consensus.cc:1261] T f73cf4b21a754e75b2af936d7402def2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:42.480808 13466 raft_consensus.cc:2932] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.484669 13079 raft_consensus.cc:1261] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:42.457059 13460 consensus_queue.cc:1035] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20260808 20:13:42.517313 12039 raft_consensus.cc:1261] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:42.511499 13389 consensus_queue.cc:1035] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260808 20:13:42.520153 13438 consensus_queue.cc:1035] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:13:42.525837 13439 consensus_queue.cc:1035] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260808 20:13:42.528929 12039 raft_consensus.cc:2932] T 4e393e90875847809e1ef94179c5b4ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.535379 13078 raft_consensus.cc:2932] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
W20260808 20:13:42.567965 13461 log.cc:924] Time spent T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260808 20:13:42.569334 12245 raft_consensus.cc:1261] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:42.571352 13439 consensus_queue.cc:1035] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260808 20:13:42.579859 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 100299us
I20260808 20:13:42.586568 13382 raft_consensus.cc:2932] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.598390 13438 raft_consensus.cc:2932] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.602237 13080 raft_consensus.cc:2932] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.620558 12146 raft_consensus.cc:2932] T 72e59eff611e4cc693407979ebe0857c P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.609923 12039 raft_consensus.cc:2932] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.744261 13440 raft_consensus.cc:2932] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.767516 11894 catalog_manager.cc:5495] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.787684 12251 raft_consensus.cc:2932] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.788766 12249 raft_consensus.cc:2932] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.756589 13077 raft_consensus.cc:2932] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.797710 13078 raft_consensus.cc:2932] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.802789 12045 raft_consensus.cc:2932] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.815584 12146 raft_consensus.cc:2932] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
W20260808 20:13:42.833189 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 50718.9us
I20260808 20:13:42.839074 11896 catalog_manager.cc:5495] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:42.841391 11896 catalog_manager.cc:5495] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:42.844542 11896 catalog_manager.cc:5495] T 72e59eff611e4cc693407979ebe0857c P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.848424 12038 raft_consensus.cc:2932] T f73cf4b21a754e75b2af936d7402def2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.893672 12140 consensus_queue.cc:237] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:42.894218 12248 consensus_queue.cc:237] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:42.895107 12146 consensus_queue.cc:237] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:42.903074 13077 raft_consensus.cc:1261] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:42.905220 13080 raft_consensus.cc:1261] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:42.905413 13386 consensus_queue.cc:1035] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 20:13:42.906896 12146 raft_consensus.cc:1261] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:42.907078 13080 raft_consensus.cc:1261] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:42.909070 13466 consensus_queue.cc:1035] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:42.910780 13466 consensus_queue.cc:1035] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:42.908785 12249 raft_consensus.cc:1261] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:42.911448 13386 consensus_queue.cc:1035] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 20:13:42.917924 13474 raft_consensus.cc:2932] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.919806 13077 raft_consensus.cc:2932] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.930609 12146 raft_consensus.cc:2932] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.934962 13450 raft_consensus.cc:2932] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.941032 13079 raft_consensus.cc:2932] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.938045 13078 raft_consensus.cc:2932] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.952245 13438 consensus_queue.cc:1035] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:42.952879 12251 consensus_queue.cc:237] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:42.955963 12245 raft_consensus.cc:2932] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.961063 12045 raft_consensus.cc:1261] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:42.962216 13076 raft_consensus.cc:1261] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:42.962391 13382 consensus_queue.cc:1035] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:42.938503 13389 raft_consensus.cc:2932] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.967566 13382 consensus_queue.cc:1035] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:42.975383 13460 raft_consensus.cc:2932] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.983702 11889 catalog_manager.cc:5495] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:42.987309 12045 raft_consensus.cc:1261] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:42.994431 11872 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 4e393e90875847809e1ef94179c5b4ef with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:42.994566 11875 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet fd0cef71281c44f6b3fd2afe876735f8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:43.006983 13079 raft_consensus.cc:2932] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:43.005865 11897 catalog_manager.cc:5495] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:43.021762 12235 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0cef71281c44f6b3fd2afe876735f8 with delete_type TABLET_DATA_TOMBSTONED (TS b9d94f35ea15421aa1d721201c782d8a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44376
I20260808 20:13:43.027457 13497 tablet_replica.cc:330] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a: stopping tablet replica
I20260808 20:13:43.030141 11889 catalog_manager.cc:5495] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:43.032188 11889 catalog_manager.cc:5495] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:43.032871 12022 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e393e90875847809e1ef94179c5b4ef with delete_type TABLET_DATA_TOMBSTONED (TS 4cd46f13bceb4c3c9ef71e755e17b995 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53014
I20260808 20:13:43.044677 11875 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f73cf4b21a754e75b2af936d7402def2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:43.046062 11872 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 72e59eff611e4cc693407979ebe0857c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:43.060201 13497 raft_consensus.cc:2229] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:43.069169 12035 tablet_service.cc:1514] Processing DeleteTablet for tablet f73cf4b21a754e75b2af936d7402def2 with delete_type TABLET_DATA_TOMBSTONED (TS 4cd46f13bceb4c3c9ef71e755e17b995 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53014
I20260808 20:13:43.070083 12131 tablet_service.cc:1514] Processing DeleteTablet for tablet 72e59eff611e4cc693407979ebe0857c with delete_type TABLET_DATA_TOMBSTONED (TS 78b9263889cd4f1582578a3aa4f644d9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53148
I20260808 20:13:43.068073 12038 raft_consensus.cc:2932] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:43.078572 13497 raft_consensus.cc:2258] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:43.078398 13389 consensus_queue.cc:1035] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:43.090956 12045 raft_consensus.cc:2932] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:43.096257 13497 ts_tablet_manager.cc:1905] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:43.099673 13498 tablet_replica.cc:330] T 4e393e90875847809e1ef94179c5b4ef P 4cd46f13bceb4c3c9ef71e755e17b995: stopping tablet replica
I20260808 20:13:43.101003 13498 raft_consensus.cc:2229] T 4e393e90875847809e1ef94179c5b4ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:43.102032 13498 raft_consensus.cc:2258] T 4e393e90875847809e1ef94179c5b4ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:43.126603 13505 tablet_replica.cc:330] T 72e59eff611e4cc693407979ebe0857c P 78b9263889cd4f1582578a3aa4f644d9: stopping tablet replica
I20260808 20:13:43.129642 13505 raft_consensus.cc:2229] T 72e59eff611e4cc693407979ebe0857c P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:43.131336 13498 ts_tablet_manager.cc:1905] T 4e393e90875847809e1ef94179c5b4ef P 4cd46f13bceb4c3c9ef71e755e17b995: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:43.132043 13505 raft_consensus.cc:2258] T 72e59eff611e4cc693407979ebe0857c P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:43.137569 13505 ts_tablet_manager.cc:1905] T 72e59eff611e4cc693407979ebe0857c P 78b9263889cd4f1582578a3aa4f644d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:43.150403 13497 ts_tablet_manager.cc:1918] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:43.151010 13497 log.cc:1196] T fd0cef71281c44f6b3fd2afe876735f8 P b9d94f35ea15421aa1d721201c782d8a: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-2-root/wals/fd0cef71281c44f6b3fd2afe876735f8
I20260808 20:13:43.152791 11872 catalog_manager.cc:4841] TS b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): tablet fd0cef71281c44f6b3fd2afe876735f8 (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:43.158231 13505 ts_tablet_manager.cc:1918] T 72e59eff611e4cc693407979ebe0857c P 78b9263889cd4f1582578a3aa4f644d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:43.158674 13505 log.cc:1196] T 72e59eff611e4cc693407979ebe0857c P 78b9263889cd4f1582578a3aa4f644d9: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-1-root/wals/72e59eff611e4cc693407979ebe0857c
I20260808 20:13:43.159587 13498 ts_tablet_manager.cc:1918] T 4e393e90875847809e1ef94179c5b4ef P 4cd46f13bceb4c3c9ef71e755e17b995: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:43.160039 13498 log.cc:1196] T 4e393e90875847809e1ef94179c5b4ef P 4cd46f13bceb4c3c9ef71e755e17b995: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-0-root/wals/4e393e90875847809e1ef94179c5b4ef
I20260808 20:13:43.160248 11875 catalog_manager.cc:4841] TS 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): tablet 72e59eff611e4cc693407979ebe0857c (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:43.161567 13498 tablet_replica.cc:330] T f73cf4b21a754e75b2af936d7402def2 P 4cd46f13bceb4c3c9ef71e755e17b995: stopping tablet replica
I20260808 20:13:43.161602 11873 catalog_manager.cc:4841] TS 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): tablet 4e393e90875847809e1ef94179c5b4ef (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:43.162789 13498 raft_consensus.cc:2229] T f73cf4b21a754e75b2af936d7402def2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:43.164208 13498 raft_consensus.cc:2258] T f73cf4b21a754e75b2af936d7402def2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:43.168672 13498 ts_tablet_manager.cc:1905] T f73cf4b21a754e75b2af936d7402def2 P 4cd46f13bceb4c3c9ef71e755e17b995: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:43.183209 13498 ts_tablet_manager.cc:1918] T f73cf4b21a754e75b2af936d7402def2 P 4cd46f13bceb4c3c9ef71e755e17b995: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:43.183493 13498 log.cc:1196] T f73cf4b21a754e75b2af936d7402def2 P 4cd46f13bceb4c3c9ef71e755e17b995: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-0-root/wals/f73cf4b21a754e75b2af936d7402def2
I20260808 20:13:43.184558 11873 catalog_manager.cc:4841] TS 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): tablet f73cf4b21a754e75b2af936d7402def2 (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:44.177704 12152 consensus_queue.cc:237] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:44.183151 12040 raft_consensus.cc:1261] T c4dc15f09b4a4470ac96054b70602187 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:44.183589 12251 raft_consensus.cc:1261] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 20:13:44.183745 12105 consensus_peers.cc:487] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4dc15f09b4a4470ac96054b70602187. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:44.184490 13438 consensus_queue.cc:1035] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:44.185835 13508 consensus_queue.cc:1035] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:44.192245 13438 raft_consensus.cc:2932] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:44.194257 12040 raft_consensus.cc:2932] T c4dc15f09b4a4470ac96054b70602187 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:44.195762 12249 raft_consensus.cc:2932] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:44.205295 12249 consensus_queue.cc:237] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:44.207733 11896 catalog_manager.cc:5495] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:44.213460 12152 raft_consensus.cc:1261] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:44.215013 13500 consensus_queue.cc:1035] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:44.221290 12040 raft_consensus.cc:1261] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 20:13:44.221724 12207 consensus_peers.cc:487] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e912d871fe214ff9bbeacaa32fd5136b. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:44.222625 13500 consensus_queue.cc:1035] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:44.232755 13460 raft_consensus.cc:2932] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:44.240535 12152 raft_consensus.cc:2932] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:44.240402 12040 raft_consensus.cc:2932] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:44.265548 12040 consensus_queue.cc:237] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:44.265587 11889 catalog_manager.cc:5495] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:44.274307 12249 raft_consensus.cc:1261] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 20:13:44.274628 12001 consensus_peers.cc:487] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7cb737cc041b47a88cab93a3e6ed8158. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:44.275527 12152 raft_consensus.cc:1261] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:44.275873 13445 consensus_queue.cc:1035] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:44.277392 13510 consensus_queue.cc:1035] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:44.284578 13445 raft_consensus.cc:2932] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:44.287533 12249 raft_consensus.cc:2932] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:44.292724 12152 raft_consensus.cc:2932] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:44.321197 12040 consensus_queue.cc:237] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:44.322184 11894 catalog_manager.cc:5495] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
W20260808 20:13:44.333863 12001 consensus_peers.cc:487] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e4bf0005d9e43c1a4adce7588e7653f. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:44.332330 12152 raft_consensus.cc:1261] T 4e4bf0005d9e43c1a4adce7588e7653f P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:44.330435 12249 raft_consensus.cc:1261] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:44.337872 13492 consensus_queue.cc:1035] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:44.347678 13526 consensus_queue.cc:1035] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:44.358393 13526 raft_consensus.cc:2932] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:44.366972 12251 raft_consensus.cc:2932] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:44.375741 12142 raft_consensus.cc:2932] T 4e4bf0005d9e43c1a4adce7588e7653f P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:44.391613 11894 catalog_manager.cc:5495] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:44.750799 13533 ts_tablet_manager.cc:927] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:13:44.752882 13533 tablet_copy_client.cc:316] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.1:40747
I20260808 20:13:44.754499 12062 tablet_copy_service.cc:140] P 4cd46f13bceb4c3c9ef71e755e17b995: Received BeginTabletCopySession request for tablet 4e4bf0005d9e43c1a4adce7588e7653f from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:49226)
I20260808 20:13:44.754947 12062 tablet_copy_service.cc:161] P 4cd46f13bceb4c3c9ef71e755e17b995: Beginning new tablet copy session on tablet 4e4bf0005d9e43c1a4adce7588e7653f from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:49226: session id = e5bd9a4616ce4acfa20dc7a002984a20-4e4bf0005d9e43c1a4adce7588e7653f
I20260808 20:13:44.761518 12062 tablet_copy_source_session.cc:205] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:44.764503 13533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e4bf0005d9e43c1a4adce7588e7653f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:44.779426 13533 tablet_copy_client.cc:791] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:44.780028 13533 tablet_copy_client.cc:655] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:44.789175 13533 tablet_copy_client.cc:531] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:44.789865 13537 ts_tablet_manager.cc:927] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:44.797151 13533 tablet_bootstrap.cc:492] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:44.798086 13537 tablet_copy_client.cc:316] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.2:35139
I20260808 20:13:44.800220 12163 tablet_copy_service.cc:140] P 78b9263889cd4f1582578a3aa4f644d9: Received BeginTabletCopySession request for tablet c4dc15f09b4a4470ac96054b70602187 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:46680)
I20260808 20:13:44.800742 12163 tablet_copy_service.cc:161] P 78b9263889cd4f1582578a3aa4f644d9: Beginning new tablet copy session on tablet c4dc15f09b4a4470ac96054b70602187 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:46680: session id = e5bd9a4616ce4acfa20dc7a002984a20-c4dc15f09b4a4470ac96054b70602187
I20260808 20:13:44.808540 12163 tablet_copy_source_session.cc:205] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:44.814033 13540 ts_tablet_manager.cc:927] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:44.814664 13537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4dc15f09b4a4470ac96054b70602187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:44.819002 13533 tablet_bootstrap.cc:492] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:44.819969 13533 tablet_bootstrap.cc:492] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:44.820801 13533 ts_tablet_manager.cc:1397] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.024s user 0.026s sys 0.000s
I20260808 20:13:44.824132 13533 raft_consensus.cc:348] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:44.825172 13533 raft_consensus.cc:729] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:44.825846 13533 consensus_queue.cc:260] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:44.829336 13533 ts_tablet_manager.cc:1428] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260808 20:13:44.832849 12062 tablet_copy_service.cc:342] P 4cd46f13bceb4c3c9ef71e755e17b995: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-4e4bf0005d9e43c1a4adce7588e7653f received from {username='slave'} at 127.0.0.1:49226
I20260808 20:13:44.833256 12062 tablet_copy_service.cc:433] P 4cd46f13bceb4c3c9ef71e755e17b995: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-4e4bf0005d9e43c1a4adce7588e7653f on tablet 4e4bf0005d9e43c1a4adce7588e7653f with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:44.833781 13540 tablet_copy_client.cc:316] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.3:38391
I20260808 20:13:44.836659 12266 tablet_copy_service.cc:140] P b9d94f35ea15421aa1d721201c782d8a: Received BeginTabletCopySession request for tablet e912d871fe214ff9bbeacaa32fd5136b from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:53030)
I20260808 20:13:44.837193 12266 tablet_copy_service.cc:161] P b9d94f35ea15421aa1d721201c782d8a: Beginning new tablet copy session on tablet e912d871fe214ff9bbeacaa32fd5136b from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:53030: session id = e5bd9a4616ce4acfa20dc7a002984a20-e912d871fe214ff9bbeacaa32fd5136b
I20260808 20:13:44.832582 13537 tablet_copy_client.cc:791] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:44.846300 13537 tablet_copy_client.cc:655] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:44.842159 13533 ts_tablet_manager.cc:927] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:13:44.850940 13537 tablet_copy_client.cc:531] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:44.851361 13533 tablet_copy_client.cc:316] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.1:40747
I20260808 20:13:44.853216 12062 tablet_copy_service.cc:140] P 4cd46f13bceb4c3c9ef71e755e17b995: Received BeginTabletCopySession request for tablet 7cb737cc041b47a88cab93a3e6ed8158 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:49226)
I20260808 20:13:44.853539 12266 tablet_copy_source_session.cc:205] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:44.854051 12062 tablet_copy_service.cc:161] P 4cd46f13bceb4c3c9ef71e755e17b995: Beginning new tablet copy session on tablet 7cb737cc041b47a88cab93a3e6ed8158 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:49226: session id = e5bd9a4616ce4acfa20dc7a002984a20-7cb737cc041b47a88cab93a3e6ed8158
I20260808 20:13:44.857021 13540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e912d871fe214ff9bbeacaa32fd5136b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:44.859524 13537 tablet_bootstrap.cc:492] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:44.859802 12062 tablet_copy_source_session.cc:205] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:44.872385 13540 tablet_copy_client.cc:791] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:44.876761 13540 tablet_copy_client.cc:655] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:44.878558 13533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cb737cc041b47a88cab93a3e6ed8158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:44.883741 13540 tablet_copy_client.cc:531] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:44.892095 13540 tablet_bootstrap.cc:492] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:44.898778 13533 tablet_copy_client.cc:791] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:44.899410 13533 tablet_copy_client.cc:655] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:44.905897 13533 tablet_copy_client.cc:531] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:44.911711 13537 tablet_bootstrap.cc:492] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:44.912575 13537 tablet_bootstrap.cc:492] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:44.913252 13537 ts_tablet_manager.cc:1397] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.054s user 0.024s sys 0.001s
I20260808 20:13:44.918571 13533 tablet_bootstrap.cc:492] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:44.918768 13537 raft_consensus.cc:348] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:44.924537 13537 raft_consensus.cc:729] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:44.925241 13537 consensus_queue.cc:260] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:44.927577 13537 ts_tablet_manager.cc:1428] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260808 20:13:44.933364 13540 tablet_bootstrap.cc:492] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:44.934239 13540 tablet_bootstrap.cc:492] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:44.938560 13540 ts_tablet_manager.cc:1397] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.047s user 0.024s sys 0.005s
I20260808 20:13:44.938953 12163 tablet_copy_service.cc:342] P 78b9263889cd4f1582578a3aa4f644d9: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-c4dc15f09b4a4470ac96054b70602187 received from {username='slave'} at 127.0.0.1:46680
I20260808 20:13:44.939368 12163 tablet_copy_service.cc:433] P 78b9263889cd4f1582578a3aa4f644d9: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-c4dc15f09b4a4470ac96054b70602187 on tablet c4dc15f09b4a4470ac96054b70602187 with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:44.941429 13540 raft_consensus.cc:348] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:44.942358 13540 raft_consensus.cc:729] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:44.943034 13540 consensus_queue.cc:260] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:44.948871 13540 ts_tablet_manager.cc:1428] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260808 20:13:44.953675 12266 tablet_copy_service.cc:342] P b9d94f35ea15421aa1d721201c782d8a: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-e912d871fe214ff9bbeacaa32fd5136b received from {username='slave'} at 127.0.0.1:53030
I20260808 20:13:44.954042 12266 tablet_copy_service.cc:433] P b9d94f35ea15421aa1d721201c782d8a: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-e912d871fe214ff9bbeacaa32fd5136b on tablet e912d871fe214ff9bbeacaa32fd5136b with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:44.964425 13533 tablet_bootstrap.cc:492] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:44.965245 13533 tablet_bootstrap.cc:492] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:44.965759 13533 ts_tablet_manager.cc:1397] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.047s user 0.021s sys 0.016s
I20260808 20:13:44.967752 13533 raft_consensus.cc:348] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:44.968577 13533 raft_consensus.cc:729] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:44.969197 13533 consensus_queue.cc:260] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:44.971495 13533 ts_tablet_manager.cc:1428] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260808 20:13:44.974534 12062 tablet_copy_service.cc:342] P 4cd46f13bceb4c3c9ef71e755e17b995: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-7cb737cc041b47a88cab93a3e6ed8158 received from {username='slave'} at 127.0.0.1:49226
I20260808 20:13:44.974946 12062 tablet_copy_service.cc:433] P 4cd46f13bceb4c3c9ef71e755e17b995: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-7cb737cc041b47a88cab93a3e6ed8158 on tablet 7cb737cc041b47a88cab93a3e6ed8158 with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:45.173007 13079 raft_consensus.cc:1203] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:45.191088 13079 raft_consensus.cc:1203] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:45.298416 13079 raft_consensus.cc:1203] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:45.345877 13079 raft_consensus.cc:1203] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:45.597082 13508 raft_consensus.cc:1050] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:45.599466 13508 consensus_queue.cc:237] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:45.607550 12251 raft_consensus.cc:1261] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:45.609014 13079 raft_consensus.cc:1261] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:45.609458 13438 consensus_queue.cc:1035] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:45.611270 13438 consensus_queue.cc:1035] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:45.628193 12045 raft_consensus.cc:1261] T c4dc15f09b4a4470ac96054b70602187 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:45.630164 13529 consensus_queue.cc:1035] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:45.635630 13438 raft_consensus.cc:2932] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:45.636613 13077 raft_consensus.cc:2932] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:45.648087 12249 raft_consensus.cc:2932] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:45.652340 12045 raft_consensus.cc:2932] T c4dc15f09b4a4470ac96054b70602187 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:45.654256 11889 catalog_manager.cc:5495] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:45.695386 12152 consensus_queue.cc:237] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:45.706075 13077 raft_consensus.cc:1261] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:45.708423 13438 consensus_queue.cc:1035] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:45.709116 12251 raft_consensus.cc:1261] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:45.712708 13558 consensus_queue.cc:1035] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:45.715524 13526 raft_consensus.cc:1050] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:45.719012 13526 consensus_queue.cc:237] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:45.725118 13079 raft_consensus.cc:2932] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:45.723490 13529 raft_consensus.cc:2932] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:45.739856 13079 raft_consensus.cc:1261] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:45.742578 13526 consensus_queue.cc:1035] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:45.743479 11897 catalog_manager.cc:5495] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:45.746865 12245 raft_consensus.cc:1261] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:45.750330 12142 raft_consensus.cc:1261] T 4e4bf0005d9e43c1a4adce7588e7653f P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:45.752152 13526 consensus_queue.cc:1035] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:45.754204 13555 consensus_queue.cc:1035] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:45.754299 12245 raft_consensus.cc:2932] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:45.762035 12022 tablet_service.cc:1514] Processing DeleteTablet for tablet c4dc15f09b4a4470ac96054b70602187 with delete_type TABLET_DATA_TOMBSTONED (TS 4cd46f13bceb4c3c9ef71e755e17b995 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53014
I20260808 20:13:45.769258 11875 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c4dc15f09b4a4470ac96054b70602187 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:45.776551 13574 tablet_replica.cc:330] T c4dc15f09b4a4470ac96054b70602187 P 4cd46f13bceb4c3c9ef71e755e17b995: stopping tablet replica
I20260808 20:13:45.777606 13574 raft_consensus.cc:2229] T c4dc15f09b4a4470ac96054b70602187 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:45.778199 13574 raft_consensus.cc:2258] T c4dc15f09b4a4470ac96054b70602187 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:45.783181 13574 ts_tablet_manager.cc:1905] T c4dc15f09b4a4470ac96054b70602187 P 4cd46f13bceb4c3c9ef71e755e17b995: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:45.783130 13510 raft_consensus.cc:2932] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:45.784268 12251 raft_consensus.cc:2932] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:45.801558 13079 raft_consensus.cc:2932] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:45.805151 13574 ts_tablet_manager.cc:1918] T c4dc15f09b4a4470ac96054b70602187 P 4cd46f13bceb4c3c9ef71e755e17b995: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:45.805619 13574 log.cc:1196] T c4dc15f09b4a4470ac96054b70602187 P 4cd46f13bceb4c3c9ef71e755e17b995: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-0-root/wals/c4dc15f09b4a4470ac96054b70602187
I20260808 20:13:45.807255 11873 catalog_manager.cc:4841] TS 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): tablet c4dc15f09b4a4470ac96054b70602187 (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:45.822036 12142 raft_consensus.cc:2932] T 4e4bf0005d9e43c1a4adce7588e7653f P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:45.823485 11897 catalog_manager.cc:5495] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:45.840595 12040 consensus_queue.cc:237] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:45.848242 13079 raft_consensus.cc:1261] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:45.849431 12245 raft_consensus.cc:1261] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:45.849341 13576 consensus_queue.cc:1035] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:45.851006 13576 consensus_queue.cc:1035] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:45.855669 13526 raft_consensus.cc:2932] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:45.857862 12251 raft_consensus.cc:2932] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:45.861452 13079 raft_consensus.cc:2932] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:45.862811 11873 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 4e4bf0005d9e43c1a4adce7588e7653f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:45.874208 11897 catalog_manager.cc:5495] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:45.887192 12131 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e4bf0005d9e43c1a4adce7588e7653f with delete_type TABLET_DATA_TOMBSTONED (TS 78b9263889cd4f1582578a3aa4f644d9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53148
I20260808 20:13:45.889606 13577 tablet_replica.cc:330] T 4e4bf0005d9e43c1a4adce7588e7653f P 78b9263889cd4f1582578a3aa4f644d9: stopping tablet replica
I20260808 20:13:45.890575 13577 raft_consensus.cc:2229] T 4e4bf0005d9e43c1a4adce7588e7653f P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:45.894234 13577 raft_consensus.cc:2258] T 4e4bf0005d9e43c1a4adce7588e7653f P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:45.898355 13577 ts_tablet_manager.cc:1905] T 4e4bf0005d9e43c1a4adce7588e7653f P 78b9263889cd4f1582578a3aa4f644d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:45.912207 13576 raft_consensus.cc:1050] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:45.912804 13577 ts_tablet_manager.cc:1918] T 4e4bf0005d9e43c1a4adce7588e7653f P 78b9263889cd4f1582578a3aa4f644d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:45.913280 13577 log.cc:1196] T 4e4bf0005d9e43c1a4adce7588e7653f P 78b9263889cd4f1582578a3aa4f644d9: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-1-root/wals/4e4bf0005d9e43c1a4adce7588e7653f
I20260808 20:13:45.914628 11875 catalog_manager.cc:4841] TS 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): tablet 4e4bf0005d9e43c1a4adce7588e7653f (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:45.914578 13576 consensus_queue.cc:237] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:45.920334 12251 raft_consensus.cc:1261] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:45.920343 13079 raft_consensus.cc:1261] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:45.921481 13510 consensus_queue.cc:1035] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:13:45.922569 13576 consensus_queue.cc:1035] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:45.927841 12142 raft_consensus.cc:1261] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:45.929323 13510 consensus_queue.cc:1035] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:45.938472 13510 raft_consensus.cc:2932] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:45.941490 13079 raft_consensus.cc:2932] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:45.960337 12245 raft_consensus.cc:2932] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:45.960461 11897 catalog_manager.cc:5495] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:45.961033 12152 raft_consensus.cc:2932] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:45.966666 13556 raft_consensus.cc:1050] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:45.969048 13556 consensus_queue.cc:237] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:45.980633 12152 raft_consensus.cc:1261] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:45.982476 13460 consensus_queue.cc:1035] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:45.986173 12038 raft_consensus.cc:1261] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:45.992094 13584 consensus_queue.cc:1035] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260808 20:13:46.005656 13079 raft_consensus.cc:1261] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:46.007105 12038 consensus_queue.cc:237] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:46.020917 12152 raft_consensus.cc:1261] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:46.022150 13576 consensus_queue.cc:1035] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 20:13:46.024402 13079 raft_consensus.cc:1261] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:46.026417 13526 consensus_queue.cc:1035] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 20:13:46.029059 13556 consensus_queue.cc:1035] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:13:46.037490 13526 raft_consensus.cc:2932] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:46.039680 13460 raft_consensus.cc:2932] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:46.041596 13080 raft_consensus.cc:2932] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:46.045635 13077 raft_consensus.cc:2932] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:46.047128 11873 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 7cb737cc041b47a88cab93a3e6ed8158 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:46.048791 12147 raft_consensus.cc:2932] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:46.057301 11894 catalog_manager.cc:5495] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:46.068102 12142 raft_consensus.cc:2932] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:46.080173 11889 catalog_manager.cc:5495] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:46.071300 12040 raft_consensus.cc:2932] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:46.091171 12235 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cb737cc041b47a88cab93a3e6ed8158 with delete_type TABLET_DATA_TOMBSTONED (TS b9d94f35ea15421aa1d721201c782d8a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44376
I20260808 20:13:46.112576 12251 consensus_queue.cc:237] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:46.118790 13591 tablet_replica.cc:330] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a: stopping tablet replica
I20260808 20:13:46.120865 13591 raft_consensus.cc:2229] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:46.124081 13591 raft_consensus.cc:2258] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:46.129012 13077 raft_consensus.cc:1261] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:46.130730 13556 consensus_queue.cc:1035] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 20:13:46.134280 12038 raft_consensus.cc:1261] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:46.138641 13556 consensus_queue.cc:1035] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:46.141134 13460 raft_consensus.cc:2932] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:46.141942 13591 ts_tablet_manager.cc:1905] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:46.146418 13077 raft_consensus.cc:2932] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:46.151655 11872 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e912d871fe214ff9bbeacaa32fd5136b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:46.156481 12040 raft_consensus.cc:2932] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:46.162065 11894 catalog_manager.cc:5495] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:46.177794 12131 tablet_service.cc:1514] Processing DeleteTablet for tablet e912d871fe214ff9bbeacaa32fd5136b with delete_type TABLET_DATA_TOMBSTONED (TS 78b9263889cd4f1582578a3aa4f644d9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53148
I20260808 20:13:46.179041 13577 tablet_replica.cc:330] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9: stopping tablet replica
I20260808 20:13:46.179237 13591 ts_tablet_manager.cc:1918] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:46.179786 13591 log.cc:1196] T 7cb737cc041b47a88cab93a3e6ed8158 P b9d94f35ea15421aa1d721201c782d8a: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-2-root/wals/7cb737cc041b47a88cab93a3e6ed8158
I20260808 20:13:46.180840 13577 raft_consensus.cc:2229] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:46.181859 13577 raft_consensus.cc:2258] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:46.184234 11872 catalog_manager.cc:4841] TS b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): tablet 7cb737cc041b47a88cab93a3e6ed8158 (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:46.191020 13577 ts_tablet_manager.cc:1905] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:46.213073 13577 ts_tablet_manager.cc:1918] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:46.213651 13577 log.cc:1196] T e912d871fe214ff9bbeacaa32fd5136b P 78b9263889cd4f1582578a3aa4f644d9: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-1-root/wals/e912d871fe214ff9bbeacaa32fd5136b
I20260808 20:13:46.215085 11875 catalog_manager.cc:4841] TS 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): tablet e912d871fe214ff9bbeacaa32fd5136b (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:47.219708 12142 consensus_queue.cc:237] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
W20260808 20:13:47.225719 12105 consensus_peers.cc:487] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5fdbdda415914f1a9652ef68d1bba7fb. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:47.226136 12249 raft_consensus.cc:1261] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:47.227311 13529 consensus_queue.cc:1035] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:47.227880 12038 raft_consensus.cc:1261] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:47.229017 13529 consensus_queue.cc:1035] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:47.234753 13529 raft_consensus.cc:2932] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:47.236390 12249 raft_consensus.cc:2932] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:47.238165 12038 raft_consensus.cc:2932] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:47.251705 12142 consensus_queue.cc:237] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:47.252346 11896 catalog_manager.cc:5495] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:47.273517 12038 raft_consensus.cc:1261] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:47.273929 12251 raft_consensus.cc:1261] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260808 20:13:47.274921 12105 consensus_peers.cc:487] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5dc2ab0712f4054883178f4b4c50928. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:47.275053 13602 consensus_queue.cc:1035] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 20:13:47.276342 13558 consensus_queue.cc:1035] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:47.290061 13558 raft_consensus.cc:2932] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:47.292215 12251 raft_consensus.cc:2932] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:47.306885 12040 raft_consensus.cc:2932] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:47.308048 12045 consensus_queue.cc:237] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:47.311712 11894 catalog_manager.cc:5495] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 2 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260808 20:13:47.339452 12001 consensus_peers.cc:487] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 926cdce6649c4ca59961cfac78392647. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:47.339577 12249 raft_consensus.cc:1261] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:47.340859 12142 raft_consensus.cc:1261] T 926cdce6649c4ca59961cfac78392647 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:47.342760 13576 consensus_queue.cc:1035] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:47.344396 13510 consensus_queue.cc:1035] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:47.356159 13615 raft_consensus.cc:2932] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:47.357570 12142 raft_consensus.cc:2932] T 926cdce6649c4ca59961cfac78392647 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:47.358430 12249 raft_consensus.cc:2932] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:47.370981 12249 consensus_queue.cc:237] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:47.375790 11894 catalog_manager.cc:5495] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260808 20:13:47.395519 12207 consensus_peers.cc:487] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 358aa11611bd4b16a062e6ced419bef8. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:47.396996 12045 raft_consensus.cc:1261] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:47.397260 12142 raft_consensus.cc:1261] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:47.398455 13556 consensus_queue.cc:1035] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:47.399401 13584 consensus_queue.cc:1035] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:47.406668 13556 raft_consensus.cc:2932] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:47.408229 12045 raft_consensus.cc:2932] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:47.408951 12142 raft_consensus.cc:2932] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:47.428282 11889 catalog_manager.cc:5495] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:47.602533 13631 ts_tablet_manager.cc:927] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:47.605433 13631 tablet_copy_client.cc:316] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.2:35139
I20260808 20:13:47.606670 12163 tablet_copy_service.cc:140] P 78b9263889cd4f1582578a3aa4f644d9: Received BeginTabletCopySession request for tablet 5fdbdda415914f1a9652ef68d1bba7fb from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:46680)
I20260808 20:13:47.607090 12163 tablet_copy_service.cc:161] P 78b9263889cd4f1582578a3aa4f644d9: Beginning new tablet copy session on tablet 5fdbdda415914f1a9652ef68d1bba7fb from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:46680: session id = e5bd9a4616ce4acfa20dc7a002984a20-5fdbdda415914f1a9652ef68d1bba7fb
I20260808 20:13:47.613834 12163 tablet_copy_source_session.cc:205] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:47.618254 13631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fdbdda415914f1a9652ef68d1bba7fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:47.632406 13631 tablet_copy_client.cc:791] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:47.632966 13631 tablet_copy_client.cc:655] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:47.636154 13631 tablet_copy_client.cc:531] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:47.644116 13631 tablet_bootstrap.cc:492] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:47.661624 13631 tablet_bootstrap.cc:492] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:47.662451 13631 tablet_bootstrap.cc:492] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:47.663093 13631 ts_tablet_manager.cc:1397] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.011s
I20260808 20:13:47.666282 13631 raft_consensus.cc:348] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:47.667110 13631 raft_consensus.cc:729] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:47.667716 13631 consensus_queue.cc:260] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:47.678484 13631 ts_tablet_manager.cc:1428] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.015s user 0.011s sys 0.000s
I20260808 20:13:47.680472 12163 tablet_copy_service.cc:342] P 78b9263889cd4f1582578a3aa4f644d9: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-5fdbdda415914f1a9652ef68d1bba7fb received from {username='slave'} at 127.0.0.1:46680
I20260808 20:13:47.680871 12163 tablet_copy_service.cc:433] P 78b9263889cd4f1582578a3aa4f644d9: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-5fdbdda415914f1a9652ef68d1bba7fb on tablet 5fdbdda415914f1a9652ef68d1bba7fb with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:47.800707 13631 ts_tablet_manager.cc:927] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:47.802255 13631 tablet_copy_client.cc:316] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.2:35139
I20260808 20:13:47.803557 12163 tablet_copy_service.cc:140] P 78b9263889cd4f1582578a3aa4f644d9: Received BeginTabletCopySession request for tablet e5dc2ab0712f4054883178f4b4c50928 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:46680)
I20260808 20:13:47.804008 12163 tablet_copy_service.cc:161] P 78b9263889cd4f1582578a3aa4f644d9: Beginning new tablet copy session on tablet e5dc2ab0712f4054883178f4b4c50928 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:46680: session id = e5bd9a4616ce4acfa20dc7a002984a20-e5dc2ab0712f4054883178f4b4c50928
I20260808 20:13:47.810775 12163 tablet_copy_source_session.cc:205] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:47.814365 13631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5dc2ab0712f4054883178f4b4c50928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:47.831712 13631 tablet_copy_client.cc:791] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:47.832297 13631 tablet_copy_client.cc:655] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:47.835599 13631 tablet_copy_client.cc:531] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:47.841900 13631 tablet_bootstrap.cc:492] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:47.865836 13631 tablet_bootstrap.cc:492] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:47.866811 13631 tablet_bootstrap.cc:492] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:47.867543 13631 ts_tablet_manager.cc:1397] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.001s
I20260808 20:13:47.869827 13631 raft_consensus.cc:348] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:47.870633 13631 raft_consensus.cc:729] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:47.871301 13631 consensus_queue.cc:260] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:47.873694 13631 ts_tablet_manager.cc:1428] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 20:13:47.880206 12163 tablet_copy_service.cc:342] P 78b9263889cd4f1582578a3aa4f644d9: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-e5dc2ab0712f4054883178f4b4c50928 received from {username='slave'} at 127.0.0.1:46680
I20260808 20:13:47.880541 12163 tablet_copy_service.cc:433] P 78b9263889cd4f1582578a3aa4f644d9: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-e5dc2ab0712f4054883178f4b4c50928 on tablet e5dc2ab0712f4054883178f4b4c50928 with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:47.959959 13631 ts_tablet_manager.cc:927] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:13:47.961486 13631 tablet_copy_client.cc:316] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.1:40747
I20260808 20:13:47.962855 12062 tablet_copy_service.cc:140] P 4cd46f13bceb4c3c9ef71e755e17b995: Received BeginTabletCopySession request for tablet 926cdce6649c4ca59961cfac78392647 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:49226)
I20260808 20:13:47.963348 12062 tablet_copy_service.cc:161] P 4cd46f13bceb4c3c9ef71e755e17b995: Beginning new tablet copy session on tablet 926cdce6649c4ca59961cfac78392647 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:49226: session id = e5bd9a4616ce4acfa20dc7a002984a20-926cdce6649c4ca59961cfac78392647
I20260808 20:13:47.970733 13643 ts_tablet_manager.cc:927] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:47.970034 12062 tablet_copy_source_session.cc:205] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:47.974833 13643 tablet_copy_client.cc:316] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.3:38391
I20260808 20:13:47.977043 12266 tablet_copy_service.cc:140] P b9d94f35ea15421aa1d721201c782d8a: Received BeginTabletCopySession request for tablet 358aa11611bd4b16a062e6ced419bef8 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:53030)
I20260808 20:13:47.977527 12266 tablet_copy_service.cc:161] P b9d94f35ea15421aa1d721201c782d8a: Beginning new tablet copy session on tablet 358aa11611bd4b16a062e6ced419bef8 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:53030: session id = e5bd9a4616ce4acfa20dc7a002984a20-358aa11611bd4b16a062e6ced419bef8
I20260808 20:13:47.977636 13631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 926cdce6649c4ca59961cfac78392647. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:47.985304 12266 tablet_copy_source_session.cc:205] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:47.988045 13643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 358aa11611bd4b16a062e6ced419bef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:47.998027 13631 tablet_copy_client.cc:791] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:47.998706 13631 tablet_copy_client.cc:655] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:48.001853 13631 tablet_copy_client.cc:531] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:48.005640 13643 tablet_copy_client.cc:791] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:48.006457 13643 tablet_copy_client.cc:655] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:48.009614 13631 tablet_bootstrap.cc:492] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:48.009727 13643 tablet_copy_client.cc:531] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:48.017216 13643 tablet_bootstrap.cc:492] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:48.032470 13631 tablet_bootstrap.cc:492] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:48.033293 13631 tablet_bootstrap.cc:492] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:48.033775 13631 ts_tablet_manager.cc:1397] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.012s
I20260808 20:13:48.035744 13631 raft_consensus.cc:348] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:48.036199 13643 tablet_bootstrap.cc:492] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:48.036547 13631 raft_consensus.cc:729] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:48.037124 13643 tablet_bootstrap.cc:492] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:48.037175 13631 consensus_queue.cc:260] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:48.037773 13643 ts_tablet_manager.cc:1397] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.009s
I20260808 20:13:48.039216 13631 ts_tablet_manager.cc:1428] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260808 20:13:48.040305 13643 raft_consensus.cc:348] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:48.041214 13643 raft_consensus.cc:729] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:48.043390 12062 tablet_copy_service.cc:342] P 4cd46f13bceb4c3c9ef71e755e17b995: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-926cdce6649c4ca59961cfac78392647 received from {username='slave'} at 127.0.0.1:49226
I20260808 20:13:48.043785 12062 tablet_copy_service.cc:433] P 4cd46f13bceb4c3c9ef71e755e17b995: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-926cdce6649c4ca59961cfac78392647 on tablet 926cdce6649c4ca59961cfac78392647 with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:48.041885 13643 consensus_queue.cc:260] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:48.044713 13079 raft_consensus.cc:1203] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:48.045852 13643 ts_tablet_manager.cc:1428] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260808 20:13:48.049127 12266 tablet_copy_service.cc:342] P b9d94f35ea15421aa1d721201c782d8a: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-358aa11611bd4b16a062e6ced419bef8 received from {username='slave'} at 127.0.0.1:53030
I20260808 20:13:48.049497 12266 tablet_copy_service.cc:433] P b9d94f35ea15421aa1d721201c782d8a: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-358aa11611bd4b16a062e6ced419bef8 on tablet 358aa11611bd4b16a062e6ced419bef8 with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:48.255142 13079 raft_consensus.cc:1203] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20260808 20:13:48.526443 13079 raft_consensus.cc:1203] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:48.556524 13079 raft_consensus.cc:1203] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:48.599913 13602 raft_consensus.cc:1050] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:48.602386 13602 consensus_queue.cc:237] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:48.609146 12249 raft_consensus.cc:1261] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:48.610859 13079 raft_consensus.cc:1261] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:48.611357 13602 consensus_queue.cc:1035] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:48.611830 12040 raft_consensus.cc:1261] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:48.612314 13595 consensus_queue.cc:1035] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:48.614171 13601 consensus_queue.cc:1035] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:48.631780 13601 raft_consensus.cc:2932] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:48.635780 12038 raft_consensus.cc:2932] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:48.641748 12249 raft_consensus.cc:2932] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:48.643600 13079 raft_consensus.cc:2932] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:48.648330 11894 catalog_manager.cc:5495] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:48.659132 13657 raft_consensus.cc:1050] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:48.665256 13657 consensus_queue.cc:237] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:48.682642 12245 raft_consensus.cc:1261] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 20:13:48.683398 13079 raft_consensus.cc:1261] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEARNER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 20:13:48.684260 13657 consensus_queue.cc:1035] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:48.684877 12038 raft_consensus.cc:1261] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 20:13:48.686283 13652 consensus_queue.cc:1035] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:48.691130 13595 consensus_queue.cc:1035] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:48.727921 13652 raft_consensus.cc:2932] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:48.733037 12038 raft_consensus.cc:2932] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:48.780447 12245 raft_consensus.cc:2932] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:48.835376 11897 catalog_manager.cc:5495] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:48.843742 13079 raft_consensus.cc:2932] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:48.863437 12147 consensus_queue.cc:237] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:48.869668 12142 consensus_queue.cc:237] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:48.874081 12038 raft_consensus.cc:1261] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:48.875423 13652 consensus_queue.cc:1035] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:48.876544 12251 raft_consensus.cc:1261] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 20:13:48.877983 13652 consensus_queue.cc:1035] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:48.881189 13077 raft_consensus.cc:1261] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 20:13:48.884321 13657 consensus_queue.cc:1035] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:48.887533 13601 raft_consensus.cc:2932] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:48.891088 13602 raft_consensus.cc:2932] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:48.883237 13079 raft_consensus.cc:1261] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:48.892794 12245 raft_consensus.cc:2932] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:48.894186 12045 raft_consensus.cc:2932] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:48.900713 13657 consensus_queue.cc:1035] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:48.903853 11875 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 5fdbdda415914f1a9652ef68d1bba7fb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:48.905578 11875 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e5dc2ab0712f4054883178f4b4c50928 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:48.903993 13077 raft_consensus.cc:2932] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:48.914022 13079 raft_consensus.cc:2932] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:48.916289 11889 catalog_manager.cc:5495] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New cstate: current_term: 2 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:48.926502 11896 catalog_manager.cc:5495] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:48.931336 12022 tablet_service.cc:1514] Processing DeleteTablet for tablet e5dc2ab0712f4054883178f4b4c50928 with delete_type TABLET_DATA_TOMBSTONED (TS 4cd46f13bceb4c3c9ef71e755e17b995 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53014
I20260808 20:13:48.934278 13678 tablet_replica.cc:330] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995: stopping tablet replica
I20260808 20:13:48.935307 13678 raft_consensus.cc:2229] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:48.936014 13678 raft_consensus.cc:2258] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:48.938621 12235 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fdbdda415914f1a9652ef68d1bba7fb with delete_type TABLET_DATA_TOMBSTONED (TS b9d94f35ea15421aa1d721201c782d8a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44376
I20260808 20:13:48.941634 13678 ts_tablet_manager.cc:1905] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:48.951356 13628 raft_consensus.cc:1050] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:48.953814 13628 consensus_queue.cc:237] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:48.960767 13678 ts_tablet_manager.cc:1918] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260808 20:13:48.961263 13678 log.cc:1196] T e5dc2ab0712f4054883178f4b4c50928 P 4cd46f13bceb4c3c9ef71e755e17b995: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-0-root/wals/e5dc2ab0712f4054883178f4b4c50928
I20260808 20:13:48.962891 11873 catalog_manager.cc:4841] TS 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): tablet e5dc2ab0712f4054883178f4b4c50928 (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:48.970476 13679 tablet_replica.cc:330] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a: stopping tablet replica
I20260808 20:13:48.971601 13679 raft_consensus.cc:2229] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:48.974795 12038 raft_consensus.cc:1261] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:48.976392 13077 raft_consensus.cc:1261] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:48.977684 12147 raft_consensus.cc:1261] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:48.978998 13679 raft_consensus.cc:2258] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:48.975911 13650 consensus_queue.cc:1035] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:48.982926 13679 ts_tablet_manager.cc:1905] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:48.980727 13630 consensus_queue.cc:1035] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260808 20:13:48.989377 13682 consensus_queue.cc:1035] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:49.011595 13679 ts_tablet_manager.cc:1918] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:49.010787 13682 raft_consensus.cc:2932] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:49.012032 13679 log.cc:1196] T 5fdbdda415914f1a9652ef68d1bba7fb P b9d94f35ea15421aa1d721201c782d8a: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-2-root/wals/5fdbdda415914f1a9652ef68d1bba7fb
I20260808 20:13:49.013935 11872 catalog_manager.cc:4841] TS b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): tablet 5fdbdda415914f1a9652ef68d1bba7fb (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:49.014037 13077 raft_consensus.cc:2932] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:49.020612 12038 raft_consensus.cc:2932] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:49.026301 12147 raft_consensus.cc:2932] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:49.033389 11894 catalog_manager.cc:5495] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:49.048591 12251 consensus_queue.cc:237] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:49.058133 12147 raft_consensus.cc:1261] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:49.059634 13077 raft_consensus.cc:1261] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:49.061000 13682 consensus_queue.cc:1035] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 20:13:49.062398 13650 consensus_queue.cc:1035] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 20:13:49.074153 12147 raft_consensus.cc:2932] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:49.075845 13079 raft_consensus.cc:2932] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:49.076572 13628 raft_consensus.cc:2932] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:49.092007 11872 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 358aa11611bd4b16a062e6ced419bef8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:49.094296 11896 catalog_manager.cc:5495] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:49.115190 12022 tablet_service.cc:1514] Processing DeleteTablet for tablet 358aa11611bd4b16a062e6ced419bef8 with delete_type TABLET_DATA_TOMBSTONED (TS 4cd46f13bceb4c3c9ef71e755e17b995 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53014
I20260808 20:13:49.116309 13678 tablet_replica.cc:330] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995: stopping tablet replica
I20260808 20:13:49.118093 13678 raft_consensus.cc:2229] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:49.119128 13678 raft_consensus.cc:2258] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:49.123013 13678 ts_tablet_manager.cc:1905] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:49.165423 13576 raft_consensus.cc:1050] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:49.168799 13576 consensus_queue.cc:237] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:49.172571 13678 ts_tablet_manager.cc:1918] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:49.173192 13678 log.cc:1196] T 358aa11611bd4b16a062e6ced419bef8 P 4cd46f13bceb4c3c9ef71e755e17b995: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-0-root/wals/358aa11611bd4b16a062e6ced419bef8
I20260808 20:13:49.174865 13079 raft_consensus.cc:1261] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:49.175420 12245 raft_consensus.cc:1261] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:49.175804 11873 catalog_manager.cc:4841] TS 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): tablet 358aa11611bd4b16a062e6ced419bef8 (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:49.176249 13510 consensus_queue.cc:1035] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:49.176554 12147 raft_consensus.cc:1261] T 926cdce6649c4ca59961cfac78392647 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:49.177825 13576 consensus_queue.cc:1035] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:49.183496 13510 consensus_queue.cc:1035] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:49.193841 13576 raft_consensus.cc:2932] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:49.195708 12147 raft_consensus.cc:2932] T 926cdce6649c4ca59961cfac78392647 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:49.205322 12251 raft_consensus.cc:2932] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:49.205865 13079 raft_consensus.cc:2932] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:49.216341 11896 catalog_manager.cc:5495] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:49.239732 12045 consensus_queue.cc:237] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:49.254967 13079 raft_consensus.cc:1261] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:49.255475 12251 raft_consensus.cc:1261] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:49.260397 13510 consensus_queue.cc:1035] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:49.263409 13576 consensus_queue.cc:1035] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:49.271224 13510 raft_consensus.cc:2932] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:49.272651 13079 raft_consensus.cc:2932] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:49.272781 12251 raft_consensus.cc:2932] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:49.283080 11873 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 926cdce6649c4ca59961cfac78392647 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:49.288728 11889 catalog_manager.cc:5495] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:49.306792 12131 tablet_service.cc:1514] Processing DeleteTablet for tablet 926cdce6649c4ca59961cfac78392647 with delete_type TABLET_DATA_TOMBSTONED (TS 78b9263889cd4f1582578a3aa4f644d9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53148
I20260808 20:13:49.312116 13694 tablet_replica.cc:330] T 926cdce6649c4ca59961cfac78392647 P 78b9263889cd4f1582578a3aa4f644d9: stopping tablet replica
I20260808 20:13:49.314102 13694 raft_consensus.cc:2229] T 926cdce6649c4ca59961cfac78392647 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:49.315156 13694 raft_consensus.cc:2258] T 926cdce6649c4ca59961cfac78392647 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:49.319572 13694 ts_tablet_manager.cc:1905] T 926cdce6649c4ca59961cfac78392647 P 78b9263889cd4f1582578a3aa4f644d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:49.340205 13694 ts_tablet_manager.cc:1918] T 926cdce6649c4ca59961cfac78392647 P 78b9263889cd4f1582578a3aa4f644d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:49.340595 13694 log.cc:1196] T 926cdce6649c4ca59961cfac78392647 P 78b9263889cd4f1582578a3aa4f644d9: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-1-root/wals/926cdce6649c4ca59961cfac78392647
I20260808 20:13:49.342094 11875 catalog_manager.cc:4841] TS 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): tablet 926cdce6649c4ca59961cfac78392647 (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:50.353196 12038 consensus_queue.cc:237] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
W20260808 20:13:50.359724 12001 consensus_peers.cc:487] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78ef4041ff114e34a9dff167914c95ef. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:50.360158 12245 raft_consensus.cc:1261] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:50.362234 13510 consensus_queue.cc:1035] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:50.363250 12142 raft_consensus.cc:1261] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:50.364969 13510 consensus_queue.cc:1035] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:50.373085 13510 raft_consensus.cc:2932] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:50.374382 12245 raft_consensus.cc:2932] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:50.377250 12142 raft_consensus.cc:2932] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:50.391438 12142 consensus_queue.cc:237] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:50.392289 11889 catalog_manager.cc:5495] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:50.399602 12245 raft_consensus.cc:1261] T 003bc27b07fa40e48c1e57a13b578981 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:50.400032 12045 raft_consensus.cc:1261] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:50.400740 13652 consensus_queue.cc:1035] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
W20260808 20:13:50.404379 12105 consensus_peers.cc:487] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 003bc27b07fa40e48c1e57a13b578981. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:50.409627 13652 consensus_queue.cc:1035] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 20:13:50.427893 13601 raft_consensus.cc:2932] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:50.430428 12245 raft_consensus.cc:2932] T 003bc27b07fa40e48c1e57a13b578981 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:50.430701 12045 raft_consensus.cc:2932] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:50.440439 12251 consensus_queue.cc:237] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:50.446393 11897 catalog_manager.cc:5495] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:50.455744 12142 raft_consensus.cc:1261] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:50.456983 12038 raft_consensus.cc:1261] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:50.458276 13628 consensus_queue.cc:1035] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260808 20:13:50.459177 12207 consensus_peers.cc:487] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67b75df4537844c2b76699830dedad62. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:50.459986 13628 consensus_queue.cc:1035] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:50.467182 13630 raft_consensus.cc:2932] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:50.476590 12142 raft_consensus.cc:2932] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:50.490563 12045 raft_consensus.cc:2932] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:50.491289 11889 catalog_manager.cc:5495] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:50.494606 12251 consensus_queue.cc:237] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
W20260808 20:13:50.528393 12207 consensus_peers.cc:487] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 237b38cfed904b8395749a75666444e9. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:50.528570 12045 raft_consensus.cc:1261] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:50.529836 12142 raft_consensus.cc:1261] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:50.529867 13630 consensus_queue.cc:1035] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:50.531647 13628 consensus_queue.cc:1035] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:50.541028 13713 raft_consensus.cc:2932] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:50.542228 12045 raft_consensus.cc:2932] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:50.556058 11889 catalog_manager.cc:5495] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:50.571848 12147 raft_consensus.cc:2932] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:50.736685 13723 ts_tablet_manager.cc:927] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:13:50.739396 13723 tablet_copy_client.cc:316] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.1:40747
I20260808 20:13:50.741066 12062 tablet_copy_service.cc:140] P 4cd46f13bceb4c3c9ef71e755e17b995: Received BeginTabletCopySession request for tablet 78ef4041ff114e34a9dff167914c95ef from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:49226)
I20260808 20:13:50.741539 12062 tablet_copy_service.cc:161] P 4cd46f13bceb4c3c9ef71e755e17b995: Beginning new tablet copy session on tablet 78ef4041ff114e34a9dff167914c95ef from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:49226: session id = e5bd9a4616ce4acfa20dc7a002984a20-78ef4041ff114e34a9dff167914c95ef
I20260808 20:13:50.749403 12062 tablet_copy_source_session.cc:205] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:50.757299 13723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78ef4041ff114e34a9dff167914c95ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:50.771410 13723 tablet_copy_client.cc:791] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:50.771965 13723 tablet_copy_client.cc:655] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:50.775586 13723 tablet_copy_client.cc:531] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:50.782757 13723 tablet_bootstrap.cc:492] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:50.799641 13723 tablet_bootstrap.cc:492] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:50.800400 13723 tablet_bootstrap.cc:492] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:50.801039 13723 ts_tablet_manager.cc:1397] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.004s
I20260808 20:13:50.803386 13723 raft_consensus.cc:348] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:50.804196 13723 raft_consensus.cc:729] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:50.804806 13723 consensus_queue.cc:260] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:50.807646 13723 ts_tablet_manager.cc:1428] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 20:13:50.810307 12062 tablet_copy_service.cc:342] P 4cd46f13bceb4c3c9ef71e755e17b995: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-78ef4041ff114e34a9dff167914c95ef received from {username='slave'} at 127.0.0.1:49226
I20260808 20:13:50.810648 12062 tablet_copy_service.cc:433] P 4cd46f13bceb4c3c9ef71e755e17b995: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-78ef4041ff114e34a9dff167914c95ef on tablet 78ef4041ff114e34a9dff167914c95ef with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:50.873286 13723 ts_tablet_manager.cc:927] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:50.878757 13723 tablet_copy_client.cc:316] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.2:35139
I20260808 20:13:50.880131 12163 tablet_copy_service.cc:140] P 78b9263889cd4f1582578a3aa4f644d9: Received BeginTabletCopySession request for tablet 003bc27b07fa40e48c1e57a13b578981 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:46680)
I20260808 20:13:50.880618 12163 tablet_copy_service.cc:161] P 78b9263889cd4f1582578a3aa4f644d9: Beginning new tablet copy session on tablet 003bc27b07fa40e48c1e57a13b578981 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:46680: session id = e5bd9a4616ce4acfa20dc7a002984a20-003bc27b07fa40e48c1e57a13b578981
I20260808 20:13:50.887781 12163 tablet_copy_source_session.cc:205] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:50.891947 13723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 003bc27b07fa40e48c1e57a13b578981. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:50.897607 13730 ts_tablet_manager.cc:927] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:50.899688 13730 tablet_copy_client.cc:316] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.3:38391
I20260808 20:13:50.901269 12266 tablet_copy_service.cc:140] P b9d94f35ea15421aa1d721201c782d8a: Received BeginTabletCopySession request for tablet 67b75df4537844c2b76699830dedad62 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:53030)
I20260808 20:13:50.901767 12266 tablet_copy_service.cc:161] P b9d94f35ea15421aa1d721201c782d8a: Beginning new tablet copy session on tablet 67b75df4537844c2b76699830dedad62 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:53030: session id = e5bd9a4616ce4acfa20dc7a002984a20-67b75df4537844c2b76699830dedad62
I20260808 20:13:50.910549 13723 tablet_copy_client.cc:791] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:50.911135 13723 tablet_copy_client.cc:655] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:50.912554 12266 tablet_copy_source_session.cc:205] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:50.915802 13730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b75df4537844c2b76699830dedad62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:50.915871 13723 tablet_copy_client.cc:531] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:50.926671 13723 tablet_bootstrap.cc:492] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:50.930370 13730 tablet_copy_client.cc:791] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:50.932284 13730 tablet_copy_client.cc:655] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:50.938174 13730 tablet_copy_client.cc:531] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:50.947714 13730 tablet_bootstrap.cc:492] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:50.952234 13723 tablet_bootstrap.cc:492] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:50.953186 13723 tablet_bootstrap.cc:492] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:50.953804 13723 ts_tablet_manager.cc:1397] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.028s user 0.015s sys 0.009s
I20260808 20:13:50.956351 13723 raft_consensus.cc:348] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:50.957260 13723 raft_consensus.cc:729] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:50.957870 13723 consensus_queue.cc:260] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:50.960547 13723 ts_tablet_manager.cc:1428] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260808 20:13:50.962247 12163 tablet_copy_service.cc:342] P 78b9263889cd4f1582578a3aa4f644d9: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-003bc27b07fa40e48c1e57a13b578981 received from {username='slave'} at 127.0.0.1:46680
I20260808 20:13:50.962607 12163 tablet_copy_service.cc:433] P 78b9263889cd4f1582578a3aa4f644d9: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-003bc27b07fa40e48c1e57a13b578981 on tablet 003bc27b07fa40e48c1e57a13b578981 with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:50.969591 13730 tablet_bootstrap.cc:492] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:50.972064 13730 tablet_bootstrap.cc:492] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:50.972741 13730 ts_tablet_manager.cc:1397] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.010s
I20260808 20:13:50.975302 13730 raft_consensus.cc:348] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:50.976128 13730 raft_consensus.cc:729] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:50.976719 13730 consensus_queue.cc:260] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:50.978905 13730 ts_tablet_manager.cc:1428] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260808 20:13:50.980594 12266 tablet_copy_service.cc:342] P b9d94f35ea15421aa1d721201c782d8a: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-67b75df4537844c2b76699830dedad62 received from {username='slave'} at 127.0.0.1:53030
I20260808 20:13:50.980985 12266 tablet_copy_service.cc:433] P b9d94f35ea15421aa1d721201c782d8a: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-67b75df4537844c2b76699830dedad62 on tablet 67b75df4537844c2b76699830dedad62 with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:50.992219 13730 ts_tablet_manager.cc:927] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:50.993703 13730 tablet_copy_client.cc:316] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.3:38391
I20260808 20:13:50.995004 12266 tablet_copy_service.cc:140] P b9d94f35ea15421aa1d721201c782d8a: Received BeginTabletCopySession request for tablet 237b38cfed904b8395749a75666444e9 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:53030)
I20260808 20:13:50.995386 12266 tablet_copy_service.cc:161] P b9d94f35ea15421aa1d721201c782d8a: Beginning new tablet copy session on tablet 237b38cfed904b8395749a75666444e9 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:53030: session id = e5bd9a4616ce4acfa20dc7a002984a20-237b38cfed904b8395749a75666444e9
I20260808 20:13:51.001663 12266 tablet_copy_source_session.cc:205] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:51.004319 13730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 237b38cfed904b8395749a75666444e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:51.022644 13730 tablet_copy_client.cc:791] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:51.023320 13730 tablet_copy_client.cc:655] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:51.028578 13730 tablet_copy_client.cc:531] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:51.039757 13730 tablet_bootstrap.cc:492] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:51.074775 13730 tablet_bootstrap.cc:492] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:51.075964 13730 tablet_bootstrap.cc:492] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:51.076592 13730 ts_tablet_manager.cc:1397] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.023s
I20260808 20:13:51.079720 13730 raft_consensus.cc:348] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:51.080633 13730 raft_consensus.cc:729] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:51.081547 13730 consensus_queue.cc:260] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:51.084257 13730 ts_tablet_manager.cc:1428] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260808 20:13:51.094503 12266 tablet_copy_service.cc:342] P b9d94f35ea15421aa1d721201c782d8a: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-237b38cfed904b8395749a75666444e9 received from {username='slave'} at 127.0.0.1:53030
I20260808 20:13:51.094906 12266 tablet_copy_service.cc:433] P b9d94f35ea15421aa1d721201c782d8a: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-237b38cfed904b8395749a75666444e9 on tablet 237b38cfed904b8395749a75666444e9 with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:51.163026 13079 raft_consensus.cc:1203] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:51.321895 13077 raft_consensus.cc:1203] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:51.415194 13077 raft_consensus.cc:1203] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:51.545713 13077 raft_consensus.cc:1203] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:51.593282 13677 raft_consensus.cc:1050] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:51.596570 13677 consensus_queue.cc:237] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:51.601954 13077 raft_consensus.cc:1261] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:51.602303 12245 raft_consensus.cc:1261] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:51.602902 12142 raft_consensus.cc:1261] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:51.603740 13510 consensus_queue.cc:1035] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:51.604687 13717 consensus_queue.cc:1035] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:51.607136 13677 consensus_queue.cc:1035] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:51.620208 13677 raft_consensus.cc:2932] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:51.632414 12251 raft_consensus.cc:2932] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:51.633257 13079 raft_consensus.cc:2932] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:51.644606 12147 raft_consensus.cc:2932] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:51.647264 11897 catalog_manager.cc:5495] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:51.698351 12040 consensus_queue.cc:237] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:51.702718 13077 raft_consensus.cc:1261] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:51.703842 12147 raft_consensus.cc:1261] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:51.703851 13510 consensus_queue.cc:1035] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 20:13:51.705379 13717 consensus_queue.cc:1035] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:51.710363 13628 raft_consensus.cc:1050] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:51.712559 13628 consensus_queue.cc:237] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:51.712827 13717 raft_consensus.cc:2932] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:51.714429 12147 raft_consensus.cc:2932] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:51.728340 11873 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 78ef4041ff114e34a9dff167914c95ef with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:51.718513 13077 raft_consensus.cc:2932] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:51.731761 12142 raft_consensus.cc:1261] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:51.733189 13738 consensus_queue.cc:1035] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:51.740092 13079 raft_consensus.cc:1261] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:51.748415 12040 raft_consensus.cc:1261] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:51.749992 13738 consensus_queue.cc:1035] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:51.750953 13716 consensus_queue.cc:1035] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:51.760402 11889 catalog_manager.cc:5495] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:51.766213 13738 raft_consensus.cc:2932] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:51.767304 12040 raft_consensus.cc:2932] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:51.772503 12235 tablet_service.cc:1514] Processing DeleteTablet for tablet 78ef4041ff114e34a9dff167914c95ef with delete_type TABLET_DATA_TOMBSTONED (TS b9d94f35ea15421aa1d721201c782d8a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44376
I20260808 20:13:51.774365 13758 tablet_replica.cc:330] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a: stopping tablet replica
I20260808 20:13:51.778378 13758 raft_consensus.cc:2229] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:51.779165 13758 raft_consensus.cc:2258] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:51.790208 13758 ts_tablet_manager.cc:1905] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:51.793377 13079 raft_consensus.cc:2932] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:51.795364 12147 raft_consensus.cc:2932] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:51.802681 11896 catalog_manager.cc:5495] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:51.832826 12245 consensus_queue.cc:237] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:51.836833 13758 ts_tablet_manager.cc:1918] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:51.838871 13079 raft_consensus.cc:1261] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:51.839231 13758 log.cc:1196] T 78ef4041ff114e34a9dff167914c95ef P b9d94f35ea15421aa1d721201c782d8a: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-2-root/wals/78ef4041ff114e34a9dff167914c95ef
I20260808 20:13:51.839850 12040 raft_consensus.cc:1261] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:51.840958 13628 consensus_queue.cc:1035] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:51.842471 13628 consensus_queue.cc:1035] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:51.841991 11872 catalog_manager.cc:4841] TS b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): tablet 78ef4041ff114e34a9dff167914c95ef (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:51.848663 13738 raft_consensus.cc:2932] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:51.849931 13079 raft_consensus.cc:2932] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:51.852680 12040 raft_consensus.cc:2932] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:51.861104 11872 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 67b75df4537844c2b76699830dedad62 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:51.874544 11897 catalog_manager.cc:5495] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:51.890983 12131 tablet_service.cc:1514] Processing DeleteTablet for tablet 67b75df4537844c2b76699830dedad62 with delete_type TABLET_DATA_TOMBSTONED (TS 78b9263889cd4f1582578a3aa4f644d9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53148
I20260808 20:13:51.893492 13760 tablet_replica.cc:330] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9: stopping tablet replica
I20260808 20:13:51.898270 13760 raft_consensus.cc:2229] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:51.900488 13760 raft_consensus.cc:2258] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:51.907480 13760 ts_tablet_manager.cc:1905] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:51.929312 13760 ts_tablet_manager.cc:1918] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:51.930009 13760 log.cc:1196] T 67b75df4537844c2b76699830dedad62 P 78b9263889cd4f1582578a3aa4f644d9: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-1-root/wals/67b75df4537844c2b76699830dedad62
I20260808 20:13:51.932569 11875 catalog_manager.cc:4841] TS 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): tablet 67b75df4537844c2b76699830dedad62 (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:51.941185 13737 raft_consensus.cc:1050] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:51.944535 13737 consensus_queue.cc:237] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:51.952878 13079 raft_consensus.cc:1261] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:51.953715 12245 raft_consensus.cc:1261] T 003bc27b07fa40e48c1e57a13b578981 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:51.954531 12040 raft_consensus.cc:1261] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:51.955164 13657 consensus_queue.cc:1035] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:13:51.956418 13601 consensus_queue.cc:1035] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:51.957258 13737 consensus_queue.cc:1035] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:13:51.974656 13601 raft_consensus.cc:2932] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:51.976572 12245 raft_consensus.cc:2932] T 003bc27b07fa40e48c1e57a13b578981 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:51.979871 12040 raft_consensus.cc:2932] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:51.984508 13077 raft_consensus.cc:2932] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:51.993053 11897 catalog_manager.cc:5495] T 003bc27b07fa40e48c1e57a13b578981 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:52.023474 12147 consensus_queue.cc:237] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:52.029644 13077 raft_consensus.cc:1261] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:52.030885 12040 raft_consensus.cc:1261] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:52.030956 13765 consensus_queue.cc:1035] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260808 20:13:52.033751 13601 consensus_queue.cc:1035] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:52.041859 13737 raft_consensus.cc:2932] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:52.046460 13077 raft_consensus.cc:2932] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:52.046907 12040 raft_consensus.cc:2932] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:52.059228 11897 catalog_manager.cc:5495] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:52.066272 11875 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 003bc27b07fa40e48c1e57a13b578981 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:52.076771 12235 tablet_service.cc:1514] Processing DeleteTablet for tablet 003bc27b07fa40e48c1e57a13b578981 with delete_type TABLET_DATA_TOMBSTONED (TS b9d94f35ea15421aa1d721201c782d8a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44376
I20260808 20:13:52.078845 13758 tablet_replica.cc:330] T 003bc27b07fa40e48c1e57a13b578981 P b9d94f35ea15421aa1d721201c782d8a: stopping tablet replica
I20260808 20:13:52.079890 13758 raft_consensus.cc:2229] T 003bc27b07fa40e48c1e57a13b578981 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:52.080629 13758 raft_consensus.cc:2258] T 003bc27b07fa40e48c1e57a13b578981 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:52.086685 13738 raft_consensus.cc:1050] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:52.090267 13738 consensus_queue.cc:237] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:52.088855 13758 ts_tablet_manager.cc:1905] T 003bc27b07fa40e48c1e57a13b578981 P b9d94f35ea15421aa1d721201c782d8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:52.129544 13077 raft_consensus.cc:1261] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:52.131069 12142 raft_consensus.cc:1261] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:52.130999 13738 consensus_queue.cc:1035] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:13:52.133086 12038 raft_consensus.cc:1261] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:52.134400 13628 consensus_queue.cc:1035] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:52.135953 13628 consensus_queue.cc:1035] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:52.136212 13758 ts_tablet_manager.cc:1918] T 003bc27b07fa40e48c1e57a13b578981 P b9d94f35ea15421aa1d721201c782d8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:52.136792 13758 log.cc:1196] T 003bc27b07fa40e48c1e57a13b578981 P b9d94f35ea15421aa1d721201c782d8a: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-2-root/wals/003bc27b07fa40e48c1e57a13b578981
I20260808 20:13:52.138684 11872 catalog_manager.cc:4841] TS b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): tablet 003bc27b07fa40e48c1e57a13b578981 (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:52.165920 13628 raft_consensus.cc:2932] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:52.169989 13077 raft_consensus.cc:2932] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:52.178701 12142 raft_consensus.cc:2932] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:52.179442 12038 raft_consensus.cc:2932] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:52.186906 11889 catalog_manager.cc:5495] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:52.291481 12245 consensus_queue.cc:237] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:52.311362 13077 raft_consensus.cc:1261] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:52.312958 12142 raft_consensus.cc:1261] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:52.314357 13628 consensus_queue.cc:1035] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:52.315258 13738 consensus_queue.cc:1035] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:52.324368 13716 raft_consensus.cc:2932] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:52.325780 13080 raft_consensus.cc:2932] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:52.327052 12142 raft_consensus.cc:2932] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:52.341950 11872 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 237b38cfed904b8395749a75666444e9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:52.347115 11896 catalog_manager.cc:5495] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:52.367503 12022 tablet_service.cc:1514] Processing DeleteTablet for tablet 237b38cfed904b8395749a75666444e9 with delete_type TABLET_DATA_TOMBSTONED (TS 4cd46f13bceb4c3c9ef71e755e17b995 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53014
I20260808 20:13:52.374740 13778 tablet_replica.cc:330] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995: stopping tablet replica
I20260808 20:13:52.376950 13778 raft_consensus.cc:2229] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:52.379364 13778 raft_consensus.cc:2258] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:52.384806 13778 ts_tablet_manager.cc:1905] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:52.402230 13778 ts_tablet_manager.cc:1918] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:52.402714 13778 log.cc:1196] T 237b38cfed904b8395749a75666444e9 P 4cd46f13bceb4c3c9ef71e755e17b995: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-0-root/wals/237b38cfed904b8395749a75666444e9
I20260808 20:13:52.405084 11873 catalog_manager.cc:4841] TS 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): tablet 237b38cfed904b8395749a75666444e9 (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:53.193423 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:53.255611 12147 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "21889249378248049ef043f0588cebc9"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:49548
I20260808 20:13:53.256331 12147 raft_consensus.cc:595] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:53.269280 12038 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:13:53.269708 12038 raft_consensus.cc:595] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Received request to transfer leadership to TS e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:53.271605 12038 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "926cdce6649c4ca59961cfac78392647"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:13:53.271970 12038 raft_consensus.cc:595] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Received request to transfer leadership to TS e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:53.273746 12038 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:13:53.274130 12038 raft_consensus.cc:595] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Received request to transfer leadership to TS b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:53.276072 12038 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:13:53.276544 12038 raft_consensus.cc:595] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Received request to transfer leadership to TS b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:53.278369 12038 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:13:53.278836 12038 raft_consensus.cc:595] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Received request to transfer leadership to TS b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:53.279686 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260808 20:13:53.280668 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:53.281332 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:53.322253 13510 raft_consensus.cc:979] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995: : Instructing follower b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:13:53.322693 13741 raft_consensus.cc:1067] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Signalling peer b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:13:53.323884 12245 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:44384
I20260808 20:13:53.324307 12245 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:53.324600 12245 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:53.330885 12245 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:53.332158 12245 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:53.332808 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:53.333043 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:53.333508 12038 raft_consensus.cc:3032] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:53.333877 12038 raft_consensus.cc:729] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:53.333585 12147 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:53.334627 12038 consensus_queue.cc:260] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:53.335903 12038 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:53.340191 12147 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:13:53.341253 12206 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:53.341603 12038 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:13:53.342196 13753 raft_consensus.cc:2781] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:53.344027 13753 raft_consensus.cc:686] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:53.344930 13753 consensus_queue.cc:237] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:53.354136 11897 catalog_manager.cc:5495] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 1 to 2, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 2 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:13:53.366938 13775 raft_consensus.cc:979] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9: : Instructing follower 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:13:53.367360 13775 raft_consensus.cc:1067] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Signalling peer 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:13:53.370186 12038 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "21889249378248049ef043f0588cebc9"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:53028
I20260808 20:13:53.370679 12038 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:53.370991 12038 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:53.379493 12038 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:53.382562 12038 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:53.386440 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:53.386623 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:53.387017 12147 raft_consensus.cc:3032] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:53.387164 12245 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:53.387866 12147 raft_consensus.cc:729] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:53.388628 12147 consensus_queue.cc:260] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:53.389922 12147 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:53.393976 12245 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:13:53.395223 11994 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:53.396076 13741 raft_consensus.cc:2781] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:53.396502 13741 raft_consensus.cc:686] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:53.397502 13741 consensus_queue.cc:237] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:53.407064 12147 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:13:53.412559 11897 catalog_manager.cc:5495] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 1 to 2, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:53.427683 13741 raft_consensus.cc:979] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995: : Instructing follower e5bd9a4616ce4acfa20dc7a002984a20 to start an election
I20260808 20:13:53.428079 13510 raft_consensus.cc:1067] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Signalling peer e5bd9a4616ce4acfa20dc7a002984a20 to start an election
I20260808 20:13:53.429745 13079 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "926cdce6649c4ca59961cfac78392647"
dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
from {username='slave'} at 127.0.0.1:53290
I20260808 20:13:53.430231 13079 raft_consensus.cc:482] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:53.430546 13079 raft_consensus.cc:3037] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:53.432600 12040 consensus_queue.cc:237] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:53.435885 13079 raft_consensus.cc:504] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:53.439146 12245 raft_consensus.cc:1261] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 20:13:53.439816 12001 consensus_peers.cc:487] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce978b5a06de44eabba785178c7a0100. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:53.439113 12142 raft_consensus.cc:1261] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:53.440512 13741 consensus_queue.cc:1035] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:53.441336 13510 consensus_queue.cc:1035] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:53.448781 13079 leader_election.cc:288] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:53.449921 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:53.450351 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:53.450532 12245 raft_consensus.cc:3037] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:53.450902 12038 raft_consensus.cc:3032] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:53.451210 12038 raft_consensus.cc:729] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:53.451925 12038 consensus_queue.cc:260] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:53.453087 12038 raft_consensus.cc:3037] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:53.457424 12245 raft_consensus.cc:2454] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:13:53.458665 13014 leader_election.cc:302] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:13:53.466974 12038 raft_consensus.cc:2454] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:13:53.475301 13804 raft_consensus.cc:2781] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:53.479506 13741 raft_consensus.cc:979] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995: : Instructing follower b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:13:53.480484 13510 raft_consensus.cc:2932] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:53.486279 13800 raft_consensus.cc:1067] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Signalling peer b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:13:53.489497 12251 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:44384
I20260808 20:13:53.489960 12251 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:53.490257 12251 raft_consensus.cc:3037] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:53.493266 12147 raft_consensus.cc:2932] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:53.500523 12245 raft_consensus.cc:2932] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:53.509488 12251 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:53.512252 12251 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:53.512558 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:53.512794 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:53.513203 12142 raft_consensus.cc:3037] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:53.513473 12038 raft_consensus.cc:3032] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:53.513782 12038 raft_consensus.cc:729] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:53.514490 12038 consensus_queue.cc:260] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:53.515820 12038 raft_consensus.cc:3037] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:53.515515 13804 raft_consensus.cc:686] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:13:53.517089 13804 consensus_queue.cc:237] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:53.524099 12142 raft_consensus.cc:2454] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:13:53.525213 12206 leader_election.cc:302] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:53.526073 13753 raft_consensus.cc:2781] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:53.526521 13753 raft_consensus.cc:686] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:53.527390 13753 consensus_queue.cc:237] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:53.530226 12038 raft_consensus.cc:2454] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:13:53.540787 12142 consensus_queue.cc:237] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:53.549839 11897 catalog_manager.cc:5495] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: term changed from 1 to 2, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4). New cstate: current_term: 2 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:53.554056 11894 catalog_manager.cc:5495] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:53.555421 12245 raft_consensus.cc:1261] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:53.559890 13784 consensus_queue.cc:1035] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:53.566888 12038 raft_consensus.cc:1261] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:53.572376 13784 consensus_queue.cc:1035] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 20:13:53.575122 11897 catalog_manager.cc:5495] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 1 to 2, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 2 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
W20260808 20:13:53.579270 12105 consensus_peers.cc:487] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e020c14c3aeb419493cd543071ed7801. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:53.581830 13775 raft_consensus.cc:2932] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:53.583488 12245 raft_consensus.cc:2932] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:53.589437 12038 raft_consensus.cc:2932] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:53.598814 12040 consensus_queue.cc:237] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:53.615669 12251 raft_consensus.cc:1261] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:53.613920 11894 catalog_manager.cc:5495] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:53.618005 13741 consensus_queue.cc:1035] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:53.619154 12147 raft_consensus.cc:1261] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:53.620816 13510 consensus_queue.cc:1035] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260808 20:13:53.623526 12001 consensus_peers.cc:487] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d977879a1ef46db915d4b277255a08c. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:53.637847 13812 raft_consensus.cc:2932] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:53.640339 12147 raft_consensus.cc:2932] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:53.642436 12251 raft_consensus.cc:2932] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:53.653043 12147 consensus_queue.cc:237] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:53.656169 11894 catalog_manager.cc:5495] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:53.663657 12251 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 20:13:53.663805 12105 consensus_peers.cc:487] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0f516c855714940a41eac002203e145. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:53.665030 13784 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:53.665966 12038 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:53.667135 13775 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:53.677471 13784 raft_consensus.cc:2932] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:53.679963 12251 raft_consensus.cc:2932] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:53.684628 12038 raft_consensus.cc:2932] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:53.688496 13800 raft_consensus.cc:979] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995: : Instructing follower e5bd9a4616ce4acfa20dc7a002984a20 to start an election
I20260808 20:13:53.688964 13800 raft_consensus.cc:1067] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Signalling peer e5bd9a4616ce4acfa20dc7a002984a20 to start an election
I20260808 20:13:53.693674 13079 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f"
dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
from {username='slave'} at 127.0.0.1:53290
I20260808 20:13:53.694172 13079 raft_consensus.cc:482] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:53.694497 13079 raft_consensus.cc:3037] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:53.700441 13079 raft_consensus.cc:504] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:53.707103 13079 leader_election.cc:288] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:53.707578 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:53.709100 12040 raft_consensus.cc:3032] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:53.713119 12040 raft_consensus.cc:729] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:53.713820 12040 consensus_queue.cc:260] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:53.715113 12040 raft_consensus.cc:3037] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:53.708038 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:53.712550 13800 raft_consensus.cc:979] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995: : Instructing follower b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:13:53.718591 13800 raft_consensus.cc:1067] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Signalling peer b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:13:53.712601 11897 catalog_manager.cc:5495] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:53.720129 12245 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:44384
I20260808 20:13:53.720603 12245 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:53.720984 12245 raft_consensus.cc:3037] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:53.723771 12040 raft_consensus.cc:2454] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:13:53.718148 12251 raft_consensus.cc:3037] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:53.724866 13015 leader_election.cc:302] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:13:53.725791 13804 raft_consensus.cc:2781] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:53.726226 13804 raft_consensus.cc:686] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:13:53.726976 13804 consensus_queue.cc:237] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:53.730398 12245 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:53.731227 12251 raft_consensus.cc:2454] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:13:53.732972 12245 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:53.733651 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:53.734215 12040 raft_consensus.cc:3032] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:53.734498 12040 raft_consensus.cc:729] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:53.734604 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:53.735132 12147 raft_consensus.cc:3037] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:53.735443 12040 consensus_queue.cc:260] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:53.736696 12040 raft_consensus.cc:3037] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:53.742785 12040 raft_consensus.cc:2454] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:13:53.742785 12147 raft_consensus.cc:2454] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:13:53.744145 12206 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:53.745083 13822 raft_consensus.cc:2781] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:53.745487 13822 raft_consensus.cc:686] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:53.746310 13822 consensus_queue.cc:237] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:53.750962 11894 catalog_manager.cc:5495] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: term changed from 1 to 2, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4). New cstate: current_term: 2 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:53.762665 11896 catalog_manager.cc:5495] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 1 to 2, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 2 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:13:53.783198 12040 raft_consensus.cc:1261] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:53.784866 13822 consensus_queue.cc:1035] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:53.807231 12147 raft_consensus.cc:1261] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:53.809487 13753 consensus_queue.cc:1035] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:53.860163 12245 raft_consensus.cc:1261] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:53.861337 13800 consensus_queue.cc:1035] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:53.873929 12152 raft_consensus.cc:1261] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:53.875099 13800 consensus_queue.cc:1035] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:53.938175 12147 raft_consensus.cc:1261] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:53.939512 13822 consensus_queue.cc:1035] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:53.950153 12040 raft_consensus.cc:1261] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:53.951287 13753 consensus_queue.cc:1035] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:53.986625 13847 ts_tablet_manager.cc:927] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:13:53.989248 13847 tablet_copy_client.cc:316] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.1:40747
I20260808 20:13:53.989238 12251 raft_consensus.cc:1261] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:13:53.990556 13804 consensus_queue.cc:1035] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:13:53.991006 12062 tablet_copy_service.cc:140] P 4cd46f13bceb4c3c9ef71e755e17b995: Received BeginTabletCopySession request for tablet ce978b5a06de44eabba785178c7a0100 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:49226)
I20260808 20:13:53.991569 12062 tablet_copy_service.cc:161] P 4cd46f13bceb4c3c9ef71e755e17b995: Beginning new tablet copy session on tablet ce978b5a06de44eabba785178c7a0100 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:49226: session id = e5bd9a4616ce4acfa20dc7a002984a20-ce978b5a06de44eabba785178c7a0100
I20260808 20:13:54.000942 12062 tablet_copy_source_session.cc:205] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:54.004366 13847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce978b5a06de44eabba785178c7a0100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:54.007174 12040 raft_consensus.cc:1261] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:13:54.009001 13850 consensus_queue.cc:1035] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:13:54.023092 13847 tablet_copy_client.cc:791] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:54.024451 13847 tablet_copy_client.cc:655] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:54.033483 13847 tablet_copy_client.cc:531] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:54.042910 13847 tablet_bootstrap.cc:492] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:54.053681 13854 ts_tablet_manager.cc:927] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:54.094805 13854 tablet_copy_client.cc:316] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.2:35139
I20260808 20:13:54.096655 12163 tablet_copy_service.cc:140] P 78b9263889cd4f1582578a3aa4f644d9: Received BeginTabletCopySession request for tablet e020c14c3aeb419493cd543071ed7801 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:46680)
I20260808 20:13:54.097226 12163 tablet_copy_service.cc:161] P 78b9263889cd4f1582578a3aa4f644d9: Beginning new tablet copy session on tablet e020c14c3aeb419493cd543071ed7801 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:46680: session id = e5bd9a4616ce4acfa20dc7a002984a20-e020c14c3aeb419493cd543071ed7801
I20260808 20:13:54.108260 12163 tablet_copy_source_session.cc:205] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:54.124039 12040 raft_consensus.cc:1261] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:13:54.124356 13854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e020c14c3aeb419493cd543071ed7801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:54.126010 13847 tablet_bootstrap.cc:492] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:54.126953 13847 tablet_bootstrap.cc:492] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:54.127686 13847 ts_tablet_manager.cc:1397] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.085s user 0.039s sys 0.038s
I20260808 20:13:54.127542 13850 consensus_queue.cc:1035] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:13:54.131397 13847 raft_consensus.cc:348] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:54.132355 13847 raft_consensus.cc:729] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:54.133124 13847 consensus_queue.cc:260] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:54.135856 13847 ts_tablet_manager.cc:1428] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260808 20:13:54.137192 12062 tablet_copy_service.cc:342] P 4cd46f13bceb4c3c9ef71e755e17b995: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-ce978b5a06de44eabba785178c7a0100 received from {username='slave'} at 127.0.0.1:49226
I20260808 20:13:54.137494 12062 tablet_copy_service.cc:433] P 4cd46f13bceb4c3c9ef71e755e17b995: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-ce978b5a06de44eabba785178c7a0100 on tablet ce978b5a06de44eabba785178c7a0100 with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:54.143002 13847 ts_tablet_manager.cc:927] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:54.147214 13854 tablet_copy_client.cc:791] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:54.147756 13854 tablet_copy_client.cc:655] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:54.151093 13847 tablet_copy_client.cc:316] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.2:35139
I20260808 20:13:54.153268 12245 raft_consensus.cc:1261] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:13:54.154822 13804 consensus_queue.cc:1035] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:13:54.157799 12163 tablet_copy_service.cc:140] P 78b9263889cd4f1582578a3aa4f644d9: Received BeginTabletCopySession request for tablet c0f516c855714940a41eac002203e145 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:46680)
I20260808 20:13:54.158314 12163 tablet_copy_service.cc:161] P 78b9263889cd4f1582578a3aa4f644d9: Beginning new tablet copy session on tablet c0f516c855714940a41eac002203e145 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:46680: session id = e5bd9a4616ce4acfa20dc7a002984a20-c0f516c855714940a41eac002203e145
I20260808 20:13:54.160055 13854 tablet_copy_client.cc:531] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:54.183795 12163 tablet_copy_source_session.cc:205] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:54.190644 13854 tablet_bootstrap.cc:492] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:54.201874 12142 raft_consensus.cc:1261] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:54.203318 13753 consensus_queue.cc:1035] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:54.223217 13863 ts_tablet_manager.cc:927] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
W20260808 20:13:54.223601 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 57292.6us
I20260808 20:13:54.227782 12040 raft_consensus.cc:1261] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:54.230505 13847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0f516c855714940a41eac002203e145. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:54.233115 13753 consensus_queue.cc:1035] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:54.283032 13847 tablet_copy_client.cc:791] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:54.283749 13847 tablet_copy_client.cc:655] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:54.292223 13847 tablet_copy_client.cc:531] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:54.283095 13863 tablet_copy_client.cc:316] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.1:40747
I20260808 20:13:54.296006 12062 tablet_copy_service.cc:140] P 4cd46f13bceb4c3c9ef71e755e17b995: Received BeginTabletCopySession request for tablet 2d977879a1ef46db915d4b277255a08c from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:49226)
I20260808 20:13:54.296509 12062 tablet_copy_service.cc:161] P 4cd46f13bceb4c3c9ef71e755e17b995: Beginning new tablet copy session on tablet 2d977879a1ef46db915d4b277255a08c from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:49226: session id = e5bd9a4616ce4acfa20dc7a002984a20-2d977879a1ef46db915d4b277255a08c
I20260808 20:13:54.301923 13854 tablet_bootstrap.cc:492] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:54.302814 13854 tablet_bootstrap.cc:492] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:54.303460 13854 ts_tablet_manager.cc:1397] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.113s user 0.024s sys 0.023s
I20260808 20:13:54.304991 12062 tablet_copy_source_session.cc:205] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:54.306713 13854 raft_consensus.cc:348] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:54.307752 13863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d977879a1ef46db915d4b277255a08c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:54.307823 13854 raft_consensus.cc:729] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:54.308626 13854 consensus_queue.cc:260] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:54.311282 13847 tablet_bootstrap.cc:492] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:54.312465 13854 ts_tablet_manager.cc:1428] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260808 20:13:54.314446 12163 tablet_copy_service.cc:342] P 78b9263889cd4f1582578a3aa4f644d9: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-e020c14c3aeb419493cd543071ed7801 received from {username='slave'} at 127.0.0.1:46680
I20260808 20:13:54.314802 12163 tablet_copy_service.cc:433] P 78b9263889cd4f1582578a3aa4f644d9: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-e020c14c3aeb419493cd543071ed7801 on tablet e020c14c3aeb419493cd543071ed7801 with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:54.315229 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:54.325326 13863 tablet_copy_client.cc:791] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:54.325917 13863 tablet_copy_client.cc:655] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:54.364670 13847 tablet_bootstrap.cc:492] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:54.354398 13863 tablet_copy_client.cc:531] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:54.367261 13847 tablet_bootstrap.cc:492] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:54.367933 13847 ts_tablet_manager.cc:1397] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.057s user 0.024s sys 0.009s
I20260808 20:13:54.368407 12147 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:49548
I20260808 20:13:54.369009 12147 raft_consensus.cc:595] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:54.370250 13847 raft_consensus.cc:348] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:54.371097 13847 raft_consensus.cc:729] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:54.371717 13847 consensus_queue.cc:260] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:54.373975 13847 ts_tablet_manager.cc:1428] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 20:13:54.376818 12045 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:13:54.377343 12045 raft_consensus.cc:595] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Received request to transfer leadership to TS e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:54.377736 12163 tablet_copy_service.cc:342] P 78b9263889cd4f1582578a3aa4f644d9: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-c0f516c855714940a41eac002203e145 received from {username='slave'} at 127.0.0.1:46680
I20260808 20:13:54.378118 12163 tablet_copy_service.cc:433] P 78b9263889cd4f1582578a3aa4f644d9: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-c0f516c855714940a41eac002203e145 on tablet c0f516c855714940a41eac002203e145 with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:54.380031 13080 raft_consensus.cc:1203] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:54.380625 13863 tablet_bootstrap.cc:492] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:54.416915 13863 tablet_bootstrap.cc:492] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:54.417964 13863 tablet_bootstrap.cc:492] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:54.419004 13863 ts_tablet_manager.cc:1397] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.039s user 0.029s sys 0.008s
I20260808 20:13:54.421859 13863 raft_consensus.cc:348] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:54.422806 13863 raft_consensus.cc:729] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:54.423487 13863 consensus_queue.cc:260] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:54.425911 12251 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67b75df4537844c2b76699830dedad62"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:13:54.426188 13863 ts_tablet_manager.cc:1428] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260808 20:13:54.426447 12251 raft_consensus.cc:595] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Received request to transfer leadership to TS e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:54.428097 12062 tablet_copy_service.cc:342] P 4cd46f13bceb4c3c9ef71e755e17b995: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-2d977879a1ef46db915d4b277255a08c received from {username='slave'} at 127.0.0.1:49226
I20260808 20:13:54.428447 12062 tablet_copy_service.cc:433] P 4cd46f13bceb4c3c9ef71e755e17b995: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-2d977879a1ef46db915d4b277255a08c on tablet 2d977879a1ef46db915d4b277255a08c with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:54.429471 13844 raft_consensus.cc:979] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a: : Instructing follower e5bd9a4616ce4acfa20dc7a002984a20 to start an election
I20260808 20:13:54.429998 13780 raft_consensus.cc:1067] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Signalling peer e5bd9a4616ce4acfa20dc7a002984a20 to start an election
I20260808 20:13:54.435170 13080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "67b75df4537844c2b76699830dedad62"
dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
from {username='slave'} at 127.0.0.1:57330
I20260808 20:13:54.435638 13080 raft_consensus.cc:482] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:54.435973 13080 raft_consensus.cc:3037] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:54.437172 12147 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:49548
I20260808 20:13:54.437673 12147 raft_consensus.cc:595] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:54.442760 12147 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c0f516c855714940a41eac002203e145"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:49548
I20260808 20:13:54.443228 12147 raft_consensus.cc:595] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:54.445366 12045 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:13:54.445835 12045 raft_consensus.cc:595] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Received request to transfer leadership to TS b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:54.446770 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260808 20:13:54.447732 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:54.448127 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:54.451397 13080 raft_consensus.cc:504] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:54.453685 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:54.453928 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:54.454380 12045 raft_consensus.cc:3037] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:54.454424 12251 raft_consensus.cc:3032] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:54.454756 12251 raft_consensus.cc:729] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:54.455466 12251 consensus_queue.cc:260] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:54.456408 12251 raft_consensus.cc:3037] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:54.460580 12045 raft_consensus.cc:2454] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:13:54.463428 13015 leader_election.cc:302] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:13:54.463760 13812 raft_consensus.cc:979] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995: : Instructing follower e5bd9a4616ce4acfa20dc7a002984a20 to start an election
I20260808 20:13:54.464178 13812 raft_consensus.cc:1067] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Signalling peer e5bd9a4616ce4acfa20dc7a002984a20 to start an election
I20260808 20:13:54.464278 13804 raft_consensus.cc:2781] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:54.464721 13804 raft_consensus.cc:686] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:13:54.465647 13079 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859"
dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
from {username='slave'} at 127.0.0.1:53290
I20260808 20:13:54.466144 13079 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:54.465900 13804 consensus_queue.cc:237] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:54.466444 13079 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:54.468269 12251 raft_consensus.cc:2454] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:13:54.473568 13079 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:54.475669 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:54.475850 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:54.476222 12040 raft_consensus.cc:3032] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:54.476415 12152 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:54.476569 12040 raft_consensus.cc:729] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:54.477322 12040 consensus_queue.cc:260] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:54.478346 12040 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:54.481557 12152 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:13:54.483049 13016 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:13:54.485280 13804 raft_consensus.cc:2781] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:54.485776 13804 raft_consensus.cc:686] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:13:54.486636 13804 consensus_queue.cc:237] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:54.474924 13079 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:54.461365 13080 leader_election.cc:288] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:54.493317 13741 raft_consensus.cc:979] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995: : Instructing follower b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:13:54.493799 13741 raft_consensus.cc:1067] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Signalling peer b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:13:54.491225 11896 catalog_manager.cc:5495] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: term changed from 1 to 2, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4). New cstate: current_term: 2 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:54.497751 12040 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:13:54.500674 12251 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:44384
I20260808 20:13:54.501216 12251 raft_consensus.cc:482] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:54.501528 12251 raft_consensus.cc:3037] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:54.507135 12251 raft_consensus.cc:504] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:54.508796 12251 leader_election.cc:288] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:54.509459 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:54.510073 12045 raft_consensus.cc:3032] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:54.510380 12045 raft_consensus.cc:729] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:54.511053 12045 consensus_queue.cc:260] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:54.511660 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:54.512158 12147 raft_consensus.cc:3037] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:54.512192 12045 raft_consensus.cc:3037] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:54.518170 12147 raft_consensus.cc:2454] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:13:54.520851 12206 leader_election.cc:302] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:54.522584 13780 raft_consensus.cc:2781] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:54.522087 12045 raft_consensus.cc:2454] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:13:54.522827 11896 catalog_manager.cc:5495] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: term changed from 1 to 2, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4). New cstate: current_term: 2 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:54.524761 13780 raft_consensus.cc:686] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:54.525806 13780 consensus_queue.cc:237] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:54.539861 11896 catalog_manager.cc:5495] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 1 to 2, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 2 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:13:54.596594 13784 raft_consensus.cc:979] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9: : Instructing follower 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:13:54.597111 13775 raft_consensus.cc:1067] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Signalling peer 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:13:54.598659 12040 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c0f516c855714940a41eac002203e145"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:53028
I20260808 20:13:54.599080 12040 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:54.599331 12040 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:54.610375 12040 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:54.613423 12040 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:54.614288 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:54.614750 12251 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:54.619822 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:54.621497 12152 raft_consensus.cc:3032] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:54.621778 12152 raft_consensus.cc:729] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:54.622550 12152 consensus_queue.cc:260] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:54.624058 12152 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:54.638597 12152 raft_consensus.cc:2454] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:13:54.639865 12000 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:13:54.640530 13812 raft_consensus.cc:2781] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:54.643842 13812 raft_consensus.cc:686] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:54.644661 13812 consensus_queue.cc:237] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:54.637305 12251 raft_consensus.cc:2454] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:13:54.652055 13080 raft_consensus.cc:1203] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:54.669903 11896 catalog_manager.cc:5495] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 1 to 2, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:54.690157 12040 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c0f516c855714940a41eac002203e145"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53014
I20260808 20:13:54.690757 12040 raft_consensus.cc:595] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Received request to transfer leadership
I20260808 20:13:54.705044 12040 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c0f516c855714940a41eac002203e145"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53014
I20260808 20:13:54.705582 12040 raft_consensus.cc:595] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Received request to transfer leadership
W20260808 20:13:54.706691 11873 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for c0f516c855714940a41eac002203e145 already in progress [suppressed 2 similar messages]
I20260808 20:13:54.709450 12040 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c0f516c855714940a41eac002203e145"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53014
I20260808 20:13:54.709959 12040 raft_consensus.cc:595] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Received request to transfer leadership
W20260808 20:13:54.711784 11966 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for c0f516c855714940a41eac002203e145 already in progress
W20260808 20:13:54.712121 11966 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for c0f516c855714940a41eac002203e145 already in progress
I20260808 20:13:54.726327 13080 raft_consensus.cc:1203] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:54.739471 13784 raft_consensus.cc:979] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9: : Instructing follower 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:13:54.739935 13784 raft_consensus.cc:1067] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Signalling peer 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:13:54.742530 13775 raft_consensus.cc:979] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9: : Instructing follower 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:13:54.743018 13865 raft_consensus.cc:1067] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Signalling peer 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:13:54.744045 12040 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:53028
I20260808 20:13:54.744634 12040 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:54.745581 12039 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:53028
I20260808 20:13:54.746495 12040 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:54.746575 12039 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:54.746963 12039 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:54.753981 12039 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:54.757298 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:54.757534 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:54.757840 12147 raft_consensus.cc:3032] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:54.758129 12251 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:54.758250 12147 raft_consensus.cc:729] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:54.759059 12147 consensus_queue.cc:260] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:54.760044 12147 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:54.765970 12147 raft_consensus.cc:2454] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:13:54.766834 12000 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:13:54.767627 13800 raft_consensus.cc:2781] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:54.768072 13800 raft_consensus.cc:686] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:54.768455 12251 raft_consensus.cc:2454] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:13:54.769233 12039 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:54.769039 13800 consensus_queue.cc:237] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:54.776765 12040 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:54.778121 12040 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:54.779011 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:54.779031 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:54.779671 12251 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:54.779749 12147 raft_consensus.cc:3032] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:54.780177 12147 raft_consensus.cc:729] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:54.781029 12147 consensus_queue.cc:260] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:54.782163 12147 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:54.787264 12251 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:13:54.788275 11994 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:54.788522 12147 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:13:54.789172 13812 raft_consensus.cc:2781] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:54.789633 13812 raft_consensus.cc:686] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:54.790504 13812 consensus_queue.cc:237] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:54.807870 13741 raft_consensus.cc:1050] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:54.810616 13741 consensus_queue.cc:237] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:54.813249 11896 catalog_manager.cc:5495] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 1 to 2, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:54.823936 13080 raft_consensus.cc:1261] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:54.825779 12147 raft_consensus.cc:1261] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:54.826514 12251 raft_consensus.cc:1261] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:54.827967 13800 consensus_queue.cc:1035] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:54.829864 13800 consensus_queue.cc:1035] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:13:54.831990 13812 consensus_queue.cc:1035] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:54.860996 11896 catalog_manager.cc:5495] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 1 to 2, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:54.886502 13741 raft_consensus.cc:2932] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:54.889974 12142 raft_consensus.cc:2932] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:54.897719 13079 raft_consensus.cc:2932] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:54.899395 12147 raft_consensus.cc:1261] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:13:54.902189 13804 consensus_queue.cc:1035] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:13:54.903729 12251 raft_consensus.cc:2932] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:54.919711 11897 catalog_manager.cc:5495] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:54.944144 12040 raft_consensus.cc:1261] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:13:54.945659 13850 consensus_queue.cc:1035] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:13:54.992859 12039 consensus_queue.cc:237] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:54.999109 13079 raft_consensus.cc:1261] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:54.999337 12147 raft_consensus.cc:1261] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:55.000845 12142 raft_consensus.cc:1261] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:55.000980 13895 consensus_queue.cc:1035] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 20:13:55.002244 13780 consensus_queue.cc:1035] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:55.005437 13741 consensus_queue.cc:1035] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:55.008116 13812 raft_consensus.cc:2932] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:55.014503 13079 raft_consensus.cc:2932] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:55.018432 11873 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet ce978b5a06de44eabba785178c7a0100 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:55.031006 12142 raft_consensus.cc:2932] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:55.039669 11889 catalog_manager.cc:5495] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:55.040480 12147 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 20:13:55.041759 13895 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:55.052071 13800 raft_consensus.cc:979] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995: : Instructing follower 78b9263889cd4f1582578a3aa4f644d9 to start an election
I20260808 20:13:55.052505 13800 raft_consensus.cc:1067] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Signalling peer 78b9263889cd4f1582578a3aa4f644d9 to start an election
I20260808 20:13:55.057255 12251 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 20:13:55.059733 12039 raft_consensus.cc:1261] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:13:55.061211 12235 tablet_service.cc:1514] Processing DeleteTablet for tablet ce978b5a06de44eabba785178c7a0100 with delete_type TABLET_DATA_TOMBSTONED (TS b9d94f35ea15421aa1d721201c782d8a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44376
I20260808 20:13:55.063405 13850 consensus_queue.cc:1035] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:13:55.062960 13920 tablet_replica.cc:330] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a: stopping tablet replica
I20260808 20:13:55.065802 13741 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:55.059935 13080 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Advancing to term 2
I20260808 20:13:55.089737 12251 raft_consensus.cc:1261] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:13:55.092295 12045 raft_consensus.cc:1261] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:55.092521 13850 consensus_queue.cc:1035] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260808 20:13:55.098187 13780 consensus_queue.cc:1035] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:55.115679 12147 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c0f516c855714940a41eac002203e145"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:53176
I20260808 20:13:55.116309 12147 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:55.116688 12147 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:55.124370 13080 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEARNER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 20:13:55.130964 12147 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:55.138880 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:55.139391 12147 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:55.156114 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:55.156769 12251 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:55.136765 13895 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260808 20:13:55.166476 13920 raft_consensus.cc:2229] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:55.170686 12040 raft_consensus.cc:3032] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:13:55.171066 12040 raft_consensus.cc:729] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:55.172565 13784 raft_consensus.cc:1050] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:55.195822 12040 consensus_queue.cc:260] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:55.197363 12040 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:55.217762 13920 raft_consensus.cc:2258] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:55.223851 12251 raft_consensus.cc:2454] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:13:55.229983 12040 raft_consensus.cc:2454] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
W20260808 20:13:55.234110 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 79286.7us
I20260808 20:13:55.258630 12102 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
W20260808 20:13:55.269054 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 106544us
I20260808 20:13:55.280330 13920 ts_tablet_manager.cc:1905] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260808 20:13:55.280860 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 49173.1us
I20260808 20:13:55.270442 13784 consensus_queue.cc:237] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:55.282778 12251 raft_consensus.cc:1261] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:55.284222 13921 consensus_queue.cc:1035] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:55.291132 13930 raft_consensus.cc:2781] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:13:55.297497 13930 raft_consensus.cc:686] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:55.318068 13930 consensus_queue.cc:237] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
W20260808 20:13:55.313454 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 64221.1us
I20260808 20:13:55.343428 12140 raft_consensus.cc:1261] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:55.347797 12147 raft_consensus.cc:1261] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:55.349516 12040 raft_consensus.cc:1261] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:55.350929 13775 consensus_queue.cc:1035] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:55.371433 13918 consensus_queue.cc:1035] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260808 20:13:55.380024 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 72552.7us
I20260808 20:13:55.375849 13812 consensus_queue.cc:1035] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260808 20:13:55.385210 13800 raft_consensus.cc:1050] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:55.386086 12251 raft_consensus.cc:1261] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:55.390836 13080 raft_consensus.cc:1261] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:55.394004 13920 ts_tablet_manager.cc:1918] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:55.394584 13920 log.cc:1196] T ce978b5a06de44eabba785178c7a0100 P b9d94f35ea15421aa1d721201c782d8a: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-2-root/wals/ce978b5a06de44eabba785178c7a0100
I20260808 20:13:55.397269 11872 catalog_manager.cc:4841] TS b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): tablet ce978b5a06de44eabba785178c7a0100 (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:55.399909 13775 consensus_queue.cc:1035] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:55.387588 13800 consensus_queue.cc:237] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:55.432824 13775 consensus_queue.cc:1035] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:55.491844 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:55.539660 12152 raft_consensus.cc:1261] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:55.540536 13080 raft_consensus.cc:1261] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:55.541517 13800 consensus_queue.cc:1035] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:13:55.559369 13936 consensus_queue.cc:1035] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260808 20:13:55.560299 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 134648us
I20260808 20:13:55.571048 12245 raft_consensus.cc:1261] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:55.576850 12245 raft_consensus.cc:1261] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:55.588317 13945 raft_consensus.cc:2932] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:55.591852 12039 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:13:55.592545 12039 raft_consensus.cc:595] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Received request to transfer leadership to TS b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:55.595572 12039 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "940aa50483d844deb4031f312b4a7355"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:13:55.594887 12245 raft_consensus.cc:2932] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:55.596131 12039 raft_consensus.cc:595] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Received request to transfer leadership to TS e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:55.598790 12146 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a383bcab83014009b5698c63cdac298b"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
mode: GRACEFUL
new_leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
from {username='slave'} at 127.0.0.1:49548
I20260808 20:13:55.599428 12146 raft_consensus.cc:595] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Received request to transfer leadership to TS e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:55.601691 13800 consensus_queue.cc:1035] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:55.602313 12040 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:13:55.604297 13800 consensus_queue.cc:1035] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:55.602888 12040 raft_consensus.cc:595] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Received request to transfer leadership to TS e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:55.615983 12040 raft_consensus.cc:2932] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
W20260808 20:13:55.620939 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 189833us
I20260808 20:13:55.635818 11894 catalog_manager.cc:5495] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 2 to 3, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 3 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:55.637998 13078 raft_consensus.cc:2932] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:55.651569 13921 raft_consensus.cc:979] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995: : Instructing follower e5bd9a4616ce4acfa20dc7a002984a20 to start an election
I20260808 20:13:55.652048 13921 raft_consensus.cc:1067] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Signalling peer e5bd9a4616ce4acfa20dc7a002984a20 to start an election
I20260808 20:13:55.655407 13800 raft_consensus.cc:979] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995: : Instructing follower b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:13:55.656046 13800 raft_consensus.cc:1067] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Signalling peer b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:13:55.668550 13079 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100"
dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
from {username='slave'} at 127.0.0.1:53290
I20260808 20:13:55.670266 13079 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:55.658524 12244 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:44384
I20260808 20:13:55.670943 12244 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:55.670662 13079 raft_consensus.cc:3037] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:55.671276 12244 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:55.681066 13079 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:55.683979 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:55.684571 12043 raft_consensus.cc:3032] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:55.684827 12043 raft_consensus.cc:729] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:55.684846 13079 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:55.685608 12043 consensus_queue.cc:260] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:55.686826 12043 raft_consensus.cc:3037] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:55.693946 12043 raft_consensus.cc:2454] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:13:55.695220 13015 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:13:55.694100 13812 raft_consensus.cc:2932] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:55.697630 13808 raft_consensus.cc:2781] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:55.697103 11897 catalog_manager.cc:5495] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:55.698426 13808 raft_consensus.cc:686] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:13:55.699838 12248 raft_consensus.cc:2932] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:55.702916 13808 consensus_queue.cc:237] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:55.705761 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:55.706418 12152 raft_consensus.cc:3037] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:55.712872 12146 raft_consensus.cc:2932] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:55.715725 12245 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:13:55.716320 12245 raft_consensus.cc:595] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:55.724530 12142 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:49548
I20260808 20:13:55.725139 12142 raft_consensus.cc:595] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:55.726402 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260808 20:13:55.727938 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:55.728531 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:55.736129 13962 raft_consensus.cc:979] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a: : Instructing follower 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:13:55.728523 12152 raft_consensus.cc:2454] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:13:55.737722 13938 raft_consensus.cc:1067] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Signalling peer 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:13:55.743144 12244 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:55.743772 12043 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:53018
I20260808 20:13:55.747788 12043 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:55.748250 12043 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:55.753213 12244 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:55.753921 12043 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:55.754901 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:55.755885 12040 raft_consensus.cc:3032] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:13:55.756225 12040 raft_consensus.cc:729] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:55.756546 12043 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:55.757439 12040 consensus_queue.cc:260] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:55.758064 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:55.758682 12244 raft_consensus.cc:3032] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:55.758968 12244 raft_consensus.cc:729] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:55.759116 12040 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:55.759639 12244 consensus_queue.cc:260] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:55.760818 12244 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:55.768749 12040 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:13:55.772953 12251 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260808 20:13:55.773653 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:55.774246 12142 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:55.775493 12244 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:13:55.776278 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:55.776801 12147 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:55.777465 12205 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:55.779163 11994 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:55.781582 13938 raft_consensus.cc:2781] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:13:55.782398 13938 raft_consensus.cc:686] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:55.782379 13943 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260808 20:13:55.785241 13895 raft_consensus.cc:2781] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:55.790495 12147 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:13:55.786026 13938 consensus_queue.cc:237] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:55.791229 13895 raft_consensus.cc:686] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:55.796175 12142 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:13:55.794689 13895 consensus_queue.cc:237] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:55.809531 13078 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEARNER]: Advancing to term 3
I20260808 20:13:55.825269 12040 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260808 20:13:55.827931 13079 raft_consensus.cc:2932] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:55.829918 13943 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 20:13:55.842417 13784 raft_consensus.cc:979] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9: : Instructing follower 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:13:55.843043 13784 raft_consensus.cc:1067] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Signalling peer 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:13:55.846449 12040 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:53028
I20260808 20:13:55.846983 12040 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:55.847301 12040 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:55.852999 13078 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEARNER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260808 20:13:55.877920 13941 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 20:13:55.890666 12040 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:55.893213 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:55.893760 12140 raft_consensus.cc:3032] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:55.894065 12140 raft_consensus.cc:729] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:55.894327 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:55.894958 12251 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:55.894969 12140 consensus_queue.cc:260] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:55.896234 12140 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:55.897729 13930 raft_consensus.cc:1050] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:55.902310 12251 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:13:55.903452 11994 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:55.903658 12140 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:13:55.905440 13895 raft_consensus.cc:2781] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:55.906169 13895 raft_consensus.cc:686] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:55.907965 13895 consensus_queue.cc:237] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:55.913321 13930 consensus_queue.cc:237] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:55.920039 12040 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:55.935261 12251 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 20:13:55.935261 13078 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEARNER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 20:13:55.936718 13943 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:13:55.937956 13941 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:13:55.944446 12040 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 20:13:55.946055 13930 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:13:55.960031 12040 raft_consensus.cc:2932] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:55.959525 13942 raft_consensus.cc:2932] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 2 to 5, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:55.963501 11894 catalog_manager.cc:5495] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:55.966681 12244 raft_consensus.cc:2932] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:55.979867 11896 catalog_manager.cc:5495] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: term changed from 1 to 2, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4). New cstate: current_term: 2 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:55.989187 12147 consensus_queue.cc:237] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:55.989822 13895 raft_consensus.cc:979] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995: : Instructing follower e5bd9a4616ce4acfa20dc7a002984a20 to start an election
I20260808 20:13:55.990281 13918 raft_consensus.cc:1067] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Signalling peer e5bd9a4616ce4acfa20dc7a002984a20 to start an election
I20260808 20:13:55.991734 13078 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "940aa50483d844deb4031f312b4a7355"
dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
from {username='slave'} at 127.0.0.1:53290
I20260808 20:13:55.992185 13078 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:55.992514 13078 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:55.996873 12244 raft_consensus.cc:1261] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:55.997448 13784 raft_consensus.cc:979] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9: : Instructing follower e5bd9a4616ce4acfa20dc7a002984a20 to start an election
I20260808 20:13:55.997931 13784 raft_consensus.cc:1067] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Signalling peer e5bd9a4616ce4acfa20dc7a002984a20 to start an election
I20260808 20:13:55.998651 13784 consensus_queue.cc:1035] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:56.000548 13076 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a383bcab83014009b5698c63cdac298b"
dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
from {username='slave'} at 127.0.0.1:53296
I20260808 20:13:56.001109 13076 raft_consensus.cc:482] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:56.001428 13076 raft_consensus.cc:3037] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:56.000178 13080 raft_consensus.cc:2932] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:56.009816 13079 raft_consensus.cc:1261] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:56.013557 13078 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:56.014873 13078 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:56.017705 11889 catalog_manager.cc:5495] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 1 to 2, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:56.021145 12245 raft_consensus.cc:2932] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:56.023655 11889 catalog_manager.cc:5495] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 1 to 2, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:56.026750 13941 consensus_queue.cc:1035] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:56.029222 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:56.029834 12040 raft_consensus.cc:3032] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:56.030117 12040 raft_consensus.cc:729] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:56.019382 13076 raft_consensus.cc:504] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:56.034946 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:56.036548 12244 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:56.037966 13078 raft_consensus.cc:2932] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:56.046267 12040 consensus_queue.cc:260] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:56.047504 12040 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:56.019395 13784 raft_consensus.cc:2932] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:56.052837 11897 catalog_manager.cc:5495] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 2 to 3, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 3 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:13:56.054570 12040 raft_consensus.cc:2454] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:13:56.055569 13015 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:13:56.056023 13076 leader_election.cc:288] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:56.057189 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:56.057749 12043 raft_consensus.cc:3037] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:56.058264 13808 raft_consensus.cc:2781] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:56.058727 13808 raft_consensus.cc:686] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:13:56.059546 13808 consensus_queue.cc:237] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:56.064553 12043 raft_consensus.cc:2454] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:13:56.065722 13015 leader_election.cc:302] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:13:56.066570 13944 raft_consensus.cc:2781] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:56.070861 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:56.071421 12140 raft_consensus.cc:3032] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:56.071677 12140 raft_consensus.cc:729] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:56.072310 12140 consensus_queue.cc:260] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:56.073470 12140 raft_consensus.cc:3037] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:56.087067 11875 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e020c14c3aeb419493cd543071ed7801 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:56.090346 11897 catalog_manager.cc:5495] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index 2 to 5, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:56.087212 12140 raft_consensus.cc:2454] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:13:56.094707 12040 consensus_queue.cc:237] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:56.098115 13944 raft_consensus.cc:686] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:13:56.103183 12244 raft_consensus.cc:2454] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:13:56.100344 13944 consensus_queue.cc:237] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:56.127218 12140 consensus_queue.cc:237] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:56.121323 11896 catalog_manager.cc:5495] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index 3 to 4, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:56.138760 13079 raft_consensus.cc:1261] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:56.143164 13080 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 20:13:56.150547 13945 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
W20260808 20:13:56.237928 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 115202us
I20260808 20:13:56.238517 11896 catalog_manager.cc:5495] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: term changed from 1 to 2, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4). New cstate: current_term: 2 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:56.241283 12245 raft_consensus.cc:1261] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20260808 20:13:56.242107 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 101415us
I20260808 20:13:56.244192 12043 raft_consensus.cc:1261] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:13:56.248342 12022 tablet_service.cc:1514] Processing DeleteTablet for tablet e020c14c3aeb419493cd543071ed7801 with delete_type TABLET_DATA_TOMBSTONED (TS 4cd46f13bceb4c3c9ef71e755e17b995 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53014
I20260808 20:13:56.249486 13943 raft_consensus.cc:2932] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:56.251825 13800 consensus_queue.cc:1035] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:56.251653 13078 raft_consensus.cc:2932] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:56.259475 12248 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 20:13:56.259488 12147 raft_consensus.cc:1261] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:13:56.264935 13812 consensus_queue.cc:1035] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:56.267081 14005 consensus_queue.cc:1035] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:13:56.264909 13808 consensus_queue.cc:1035] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260808 20:13:56.272111 13945 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 20:13:56.278637 14013 tablet_replica.cc:330] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995: stopping tablet replica
I20260808 20:13:56.279994 14013 raft_consensus.cc:2229] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:56.279623 12251 raft_consensus.cc:2932] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:56.280679 14013 raft_consensus.cc:2258] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:56.283840 14013 ts_tablet_manager.cc:1905] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:56.288654 12040 raft_consensus.cc:1261] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:13:56.290397 11889 catalog_manager.cc:5495] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: term changed from 1 to 2, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4). New cstate: current_term: 2 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:56.293969 12140 raft_consensus.cc:1261] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:13:56.294231 12147 raft_consensus.cc:1261] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:56.297294 13944 consensus_queue.cc:1035] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:13:56.297562 13812 consensus_queue.cc:1035] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:56.298218 14013 ts_tablet_manager.cc:1918] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:56.298611 14013 log.cc:1196] T e020c14c3aeb419493cd543071ed7801 P 4cd46f13bceb4c3c9ef71e755e17b995: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-0-root/wals/e020c14c3aeb419493cd543071ed7801
I20260808 20:13:56.300379 13938 consensus_queue.cc:1035] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:56.318984 13812 raft_consensus.cc:2932] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:56.327592 12251 raft_consensus.cc:2932] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:56.292062 11889 catalog_manager.cc:5495] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: config changed from index 5 to 6, VOTER 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) evicted. New cstate: current_term: 3 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:56.339726 11875 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c0f516c855714940a41eac002203e145 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:56.357589 13080 raft_consensus.cc:2932] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:56.345829 11896 catalog_manager.cc:5495] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:56.375218 11873 catalog_manager.cc:4841] TS 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): tablet e020c14c3aeb419493cd543071ed7801 (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:56.376044 11873 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 2d977879a1ef46db915d4b277255a08c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:56.376415 12131 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d977879a1ef46db915d4b277255a08c with delete_type TABLET_DATA_TOMBSTONED (TS 78b9263889cd4f1582578a3aa4f644d9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53148
I20260808 20:13:56.379881 14019 tablet_replica.cc:330] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9: stopping tablet replica
I20260808 20:13:56.381105 14019 raft_consensus.cc:2229] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:56.381937 14019 raft_consensus.cc:2258] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:56.383467 12244 raft_consensus.cc:1261] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:56.386433 14019 ts_tablet_manager.cc:1905] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:56.393134 13895 consensus_queue.cc:1035] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:56.407305 14019 ts_tablet_manager.cc:1918] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:56.407843 14019 log.cc:1196] T 2d977879a1ef46db915d4b277255a08c P 78b9263889cd4f1582578a3aa4f644d9: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-1-root/wals/2d977879a1ef46db915d4b277255a08c
I20260808 20:13:56.414434 11875 catalog_manager.cc:4841] TS 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): tablet 2d977879a1ef46db915d4b277255a08c (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:56.417771 12022 tablet_service.cc:1514] Processing DeleteTablet for tablet c0f516c855714940a41eac002203e145 with delete_type TABLET_DATA_TOMBSTONED (TS 4cd46f13bceb4c3c9ef71e755e17b995 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53014
I20260808 20:13:56.418807 14013 tablet_replica.cc:330] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995: stopping tablet replica
I20260808 20:13:56.422066 14013 raft_consensus.cc:2229] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:56.422873 14013 raft_consensus.cc:2258] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:56.426616 14013 ts_tablet_manager.cc:1905] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:13:56.441289 12140 raft_consensus.cc:1261] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:56.442631 14020 consensus_queue.cc:1035] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:56.443475 14013 ts_tablet_manager.cc:1918] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20260808 20:13:56.443890 14013 log.cc:1196] T c0f516c855714940a41eac002203e145 P 4cd46f13bceb4c3c9ef71e755e17b995: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-0-root/wals/c0f516c855714940a41eac002203e145
I20260808 20:13:56.444165 12040 raft_consensus.cc:1261] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:13:56.445469 13944 consensus_queue.cc:1035] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:13:56.445737 11873 catalog_manager.cc:4841] TS 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): tablet c0f516c855714940a41eac002203e145 (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:13:56.458696 12251 raft_consensus.cc:1261] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:56.459367 12244 raft_consensus.cc:1261] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:13:56.460520 13800 consensus_queue.cc:1035] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:56.460590 13999 consensus_queue.cc:1035] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:13:56.574649 12040 raft_consensus.cc:1261] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:13:56.575647 13999 consensus_queue.cc:1035] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:13:56.585520 12140 raft_consensus.cc:1261] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:13:56.589051 13944 consensus_queue.cc:1035] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:13:56.802994 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:56.838632 12140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:49548
I20260808 20:13:56.839088 12140 raft_consensus.cc:595] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:56.841073 12140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:49548
I20260808 20:13:56.841436 12140 raft_consensus.cc:595] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:56.843267 12248 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:13:56.843637 12248 raft_consensus.cc:595] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:56.845908 12140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c0f516c855714940a41eac002203e145"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
mode: GRACEFUL
new_leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
from {username='slave'} at 127.0.0.1:49548
I20260808 20:13:56.846314 12140 raft_consensus.cc:595] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Received request to transfer leadership to TS e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:56.848186 12040 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:13:56.848675 12040 raft_consensus.cc:595] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Received request to transfer leadership to TS b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:56.850929 12147 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
mode: GRACEFUL
new_leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
from {username='slave'} at 127.0.0.1:49548
I20260808 20:13:56.851399 12147 raft_consensus.cc:595] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Received request to transfer leadership to TS e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:56.852367 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260808 20:13:56.853425 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:56.853852 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:56.887290 13967 raft_consensus.cc:979] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a: : Instructing follower 78b9263889cd4f1582578a3aa4f644d9 to start an election
I20260808 20:13:56.887774 14007 raft_consensus.cc:1067] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Signalling peer 78b9263889cd4f1582578a3aa4f644d9 to start an election
I20260808 20:13:56.889153 12147 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:53162
I20260808 20:13:56.889673 12147 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:56.889992 12147 raft_consensus.cc:3037] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:56.894248 12147 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:56.895488 12147 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:56.896131 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:56.896422 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:56.896701 12040 raft_consensus.cc:3037] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:56.896989 12248 raft_consensus.cc:3032] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:13:56.897266 12248 raft_consensus.cc:729] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:56.897891 12248 consensus_queue.cc:260] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:56.899000 12248 raft_consensus.cc:3037] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:56.903723 12040 raft_consensus.cc:2454] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:13:56.905179 13918 raft_consensus.cc:979] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995: : Instructing follower b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:13:56.905179 12248 raft_consensus.cc:2454] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:13:56.905475 12102 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:13:56.905639 13918 raft_consensus.cc:1067] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Signalling peer b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:13:56.906476 13942 raft_consensus.cc:2781] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:13:56.907155 13942 raft_consensus.cc:686] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:56.907817 12248 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:44384
I20260808 20:13:56.908355 12248 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:56.907971 13942 consensus_queue.cc:237] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:56.908682 12248 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:56.914661 12248 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:56.916277 12248 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:56.917539 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:56.918061 12045 raft_consensus.cc:3032] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:13:56.918292 12045 raft_consensus.cc:729] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:56.918905 12045 consensus_queue.cc:260] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:56.919411 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:56.920047 12147 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:56.920084 12045 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:56.922135 11889 catalog_manager.cc:5495] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 2 to 3, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 3 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:56.929137 12147 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:13:56.930400 12206 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:56.929219 12045 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:13:56.931509 14007 raft_consensus.cc:2781] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:13:56.931945 14007 raft_consensus.cc:686] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:56.932740 14007 consensus_queue.cc:237] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:56.951426 11889 catalog_manager.cc:5495] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 2 to 3, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 3 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:13:56.963311 13943 raft_consensus.cc:979] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9: : Instructing follower 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:13:56.963691 13942 raft_consensus.cc:1067] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Signalling peer 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:13:56.964970 12045 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:53028
I20260808 20:13:56.965442 12045 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:56.965786 12045 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:56.970026 12045 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:56.971261 12045 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:56.971845 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:56.972052 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:56.972419 12140 raft_consensus.cc:3032] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:56.972508 12248 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:56.972707 12140 raft_consensus.cc:729] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:56.973392 12140 consensus_queue.cc:260] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:56.974320 12140 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:56.977751 12248 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:13:56.978703 11994 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:56.979264 13800 raft_consensus.cc:2781] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:56.979516 12140 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:13:56.979612 13800 raft_consensus.cc:686] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:56.980365 13800 consensus_queue.cc:237] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:56.988590 11889 catalog_manager.cc:5495] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 1 to 2, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:57.056078 13943 raft_consensus.cc:979] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9: : Instructing follower 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:13:57.056524 13942 raft_consensus.cc:1067] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Signalling peer 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:13:57.057825 12040 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:53028
I20260808 20:13:57.058233 12040 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:57.058480 12040 raft_consensus.cc:3037] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:57.063066 12040 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:57.064924 12040 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:57.065186 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:57.065814 12140 raft_consensus.cc:3032] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:57.066308 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:57.066874 12244 raft_consensus.cc:3037] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:57.066332 12140 raft_consensus.cc:729] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:57.067854 12140 consensus_queue.cc:260] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:57.068955 12140 raft_consensus.cc:3037] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:57.072968 12244 raft_consensus.cc:2454] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:13:57.073853 11994 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:57.074434 13800 raft_consensus.cc:2781] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:57.074757 13800 raft_consensus.cc:686] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:57.074983 12140 raft_consensus.cc:2454] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:13:57.075420 13800 consensus_queue.cc:237] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:57.084622 11889 catalog_manager.cc:5495] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 1 to 2, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:57.097096 13942 raft_consensus.cc:979] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9: : Instructing follower e5bd9a4616ce4acfa20dc7a002984a20 to start an election
I20260808 20:13:57.097551 13943 raft_consensus.cc:1067] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Signalling peer e5bd9a4616ce4acfa20dc7a002984a20 to start an election
I20260808 20:13:57.098753 13080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c0f516c855714940a41eac002203e145"
dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
from {username='slave'} at 127.0.0.1:53296
I20260808 20:13:57.099148 13080 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:57.099367 13080 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:13:57.104525 13080 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:57.105896 13080 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:57.106489 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:57.106760 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:57.107025 12140 raft_consensus.cc:3032] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:13:57.107264 12244 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:13:57.107313 12140 raft_consensus.cc:729] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:57.107986 12140 consensus_queue.cc:260] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:57.108816 12140 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:13:57.111379 12244 raft_consensus.cc:2454] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:13:57.112130 13014 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:13:57.112681 14003 raft_consensus.cc:2781] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:13:57.113091 12140 raft_consensus.cc:2454] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:13:57.113126 14003 raft_consensus.cc:686] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:13:57.114017 14003 consensus_queue.cc:237] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:57.120419 11889 catalog_manager.cc:5495] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: term changed from 3 to 4, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4). New cstate: current_term: 4 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:57.250378 13943 raft_consensus.cc:979] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9: : Instructing follower e5bd9a4616ce4acfa20dc7a002984a20 to start an election
I20260808 20:13:57.250818 13942 raft_consensus.cc:1067] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Signalling peer e5bd9a4616ce4acfa20dc7a002984a20 to start an election
I20260808 20:13:57.252007 13080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2"
dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
from {username='slave'} at 127.0.0.1:53296
I20260808 20:13:57.252444 13080 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:57.252743 13080 raft_consensus.cc:3037] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:57.258363 13080 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:57.259840 13080 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:57.260443 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:57.260792 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:57.261013 12140 raft_consensus.cc:3032] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:57.261299 12244 raft_consensus.cc:3037] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:57.261325 12140 raft_consensus.cc:729] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:57.262085 12140 consensus_queue.cc:260] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:57.263028 12140 raft_consensus.cc:3037] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:57.265478 12244 raft_consensus.cc:2454] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:13:57.266283 13014 leader_election.cc:302] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:13:57.266885 14003 raft_consensus.cc:2781] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:57.267386 14003 raft_consensus.cc:686] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:13:57.268055 12140 raft_consensus.cc:2454] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:13:57.268792 14003 consensus_queue.cc:237] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:57.276794 11889 catalog_manager.cc:5495] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: term changed from 1 to 2, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4). New cstate: current_term: 2 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:57.334169 12040 raft_consensus.cc:1261] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:13:57.335227 13942 consensus_queue.cc:1035] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:57.347819 12244 raft_consensus.cc:1261] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:13:57.348820 14043 consensus_queue.cc:1035] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:57.349779 12140 raft_consensus.cc:1261] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:13:57.351068 13967 consensus_queue.cc:1035] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:57.369053 12040 raft_consensus.cc:1261] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:13:57.373852 13967 consensus_queue.cc:1035] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:57.459709 12140 raft_consensus.cc:1261] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:57.461651 13800 consensus_queue.cc:1035] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:57.476642 12251 raft_consensus.cc:1261] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:57.478657 13800 consensus_queue.cc:1035] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:57.505017 12040 consensus_queue.cc:237] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:57.510558 12251 raft_consensus.cc:1261] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 20:13:57.511509 12001 consensus_peers.cc:487] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d91cb12be034dcc9ca3038df3cc85cf. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:57.511925 13918 consensus_queue.cc:1035] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:57.511283 12147 raft_consensus.cc:1261] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:57.514047 13918 consensus_queue.cc:1035] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:57.519440 13800 raft_consensus.cc:2932] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:57.520758 12251 raft_consensus.cc:2932] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:57.523006 12147 raft_consensus.cc:2932] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:57.534946 12040 consensus_queue.cc:237] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:57.534987 11896 catalog_manager.cc:5495] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:57.543422 12140 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 20:13:57.545274 14003 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260808 20:13:57.557667 12001 consensus_peers.cc:487] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41bae2134ac343ed9e9f9c1b86703d97. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:57.561252 12140 raft_consensus.cc:1261] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:57.562497 13800 consensus_queue.cc:1035] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:57.561476 12251 raft_consensus.cc:1261] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:57.563416 12244 raft_consensus.cc:1261] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:57.563988 13918 consensus_queue.cc:1035] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:57.565482 13800 consensus_queue.cc:1035] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:57.569749 12248 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 20:13:57.571070 13808 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 20:13:57.605144 13918 raft_consensus.cc:2932] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:57.614404 12142 raft_consensus.cc:2932] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:57.622632 12140 consensus_queue.cc:237] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:57.622609 11897 catalog_manager.cc:5495] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:57.630254 12147 raft_consensus.cc:1261] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:57.633661 13800 consensus_queue.cc:1035] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:57.634135 12251 raft_consensus.cc:2932] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:57.675535 12040 raft_consensus.cc:1261] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:57.677407 12244 raft_consensus.cc:1261] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:13:57.679647 13942 consensus_queue.cc:1035] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260808 20:13:57.682142 12105 consensus_peers.cc:487] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a16032fd7a244ded8d9660c183645279. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:57.683789 14084 consensus_queue.cc:1035] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:57.687665 12142 raft_consensus.cc:1261] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:13:57.688803 13808 consensus_queue.cc:1035] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:13:57.689332 12244 raft_consensus.cc:1261] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:13:57.690536 13808 consensus_queue.cc:1035] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:13:57.702296 13942 raft_consensus.cc:2932] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:57.717507 12248 raft_consensus.cc:2932] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:57.729208 12045 raft_consensus.cc:2932] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:57.733266 12040 consensus_queue.cc:237] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
W20260808 20:13:57.740723 12001 consensus_peers.cc:487] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2a41c716cfc94652a2fc6957a1bd2a78. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:13:57.747423 11889 catalog_manager.cc:5495] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:57.751037 12245 raft_consensus.cc:1261] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 20:13:57.751972 12142 raft_consensus.cc:1261] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 20:13:57.753008 13800 consensus_queue.cc:1035] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:57.753659 14075 consensus_queue.cc:1035] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260808 20:13:57.765480 13812 raft_consensus.cc:2932] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:57.769917 12142 raft_consensus.cc:2932] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:57.771956 12248 raft_consensus.cc:2932] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:13:57.788400 11889 catalog_manager.cc:5495] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index -1 to 3, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 2 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:57.889957 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:57.936965 12040 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:13:57.937587 12040 raft_consensus.cc:595] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Received request to transfer leadership to TS e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:57.940112 14091 ts_tablet_manager.cc:927] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:13:57.940055 12245 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:13:57.940599 12245 raft_consensus.cc:595] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:57.944391 12040 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:13:57.944886 12040 raft_consensus.cc:595] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Received request to transfer leadership to TS b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:57.947937 12147 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:49548
I20260808 20:13:57.948441 12147 raft_consensus.cc:595] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:57.949581 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260808 20:13:57.951094 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:57.951620 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:57.965284 14091 tablet_copy_client.cc:316] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.1:40747
I20260808 20:13:57.966689 12062 tablet_copy_service.cc:140] P 4cd46f13bceb4c3c9ef71e755e17b995: Received BeginTabletCopySession request for tablet 41bae2134ac343ed9e9f9c1b86703d97 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:49226)
I20260808 20:13:57.967195 12062 tablet_copy_service.cc:161] P 4cd46f13bceb4c3c9ef71e755e17b995: Beginning new tablet copy session on tablet 41bae2134ac343ed9e9f9c1b86703d97 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:49226: session id = e5bd9a4616ce4acfa20dc7a002984a20-41bae2134ac343ed9e9f9c1b86703d97
I20260808 20:13:57.974501 12062 tablet_copy_source_session.cc:205] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:57.977365 14091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41bae2134ac343ed9e9f9c1b86703d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:57.992070 14091 tablet_copy_client.cc:791] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:57.992592 14091 tablet_copy_client.cc:655] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:57.992625 14093 ts_tablet_manager.cc:927] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:13:57.997073 13800 raft_consensus.cc:979] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995: : Instructing follower e5bd9a4616ce4acfa20dc7a002984a20 to start an election
I20260808 20:13:57.997488 13812 raft_consensus.cc:1067] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Signalling peer e5bd9a4616ce4acfa20dc7a002984a20 to start an election
I20260808 20:13:57.998826 13078 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef"
dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
from {username='slave'} at 127.0.0.1:53290
I20260808 20:13:57.999390 13078 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:57.999716 13078 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:58.006969 14093 tablet_copy_client.cc:316] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.1:40747
I20260808 20:13:58.006732 13078 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:58.008212 14091 tablet_copy_client.cc:531] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:58.008785 12062 tablet_copy_service.cc:140] P 4cd46f13bceb4c3c9ef71e755e17b995: Received BeginTabletCopySession request for tablet 0d91cb12be034dcc9ca3038df3cc85cf from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:49226)
I20260808 20:13:58.009393 12062 tablet_copy_service.cc:161] P 4cd46f13bceb4c3c9ef71e755e17b995: Beginning new tablet copy session on tablet 0d91cb12be034dcc9ca3038df3cc85cf from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:49226: session id = e5bd9a4616ce4acfa20dc7a002984a20-0d91cb12be034dcc9ca3038df3cc85cf
I20260808 20:13:58.010177 13078 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:58.010043 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:58.010995 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:58.011322 12045 raft_consensus.cc:3032] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:58.011591 12147 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:58.011629 12045 raft_consensus.cc:729] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:58.012598 12045 consensus_queue.cc:260] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:58.014873 12045 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:58.021672 12045 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:13:58.023389 14091 tablet_bootstrap.cc:492] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:58.024919 13015 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:13:58.026218 13944 raft_consensus.cc:2781] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:58.026683 13944 raft_consensus.cc:686] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:13:58.027513 13944 consensus_queue.cc:237] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:58.030514 12062 tablet_copy_source_session.cc:205] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:58.036756 14093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d91cb12be034dcc9ca3038df3cc85cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:58.050510 12147 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:13:58.077169 11897 catalog_manager.cc:5495] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: term changed from 1 to 2, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4). New cstate: current_term: 2 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:13:58.081247 14093 tablet_copy_client.cc:791] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:58.081815 14093 tablet_copy_client.cc:655] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:58.086198 14091 tablet_bootstrap.cc:492] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:58.087071 14091 tablet_bootstrap.cc:492] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:58.087710 14091 ts_tablet_manager.cc:1397] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.065s user 0.033s sys 0.021s
I20260808 20:13:58.090294 14091 raft_consensus.cc:348] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:58.091085 14091 raft_consensus.cc:729] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:58.091702 14091 consensus_queue.cc:260] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:58.093993 14091 ts_tablet_manager.cc:1428] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260808 20:13:58.094033 14093 tablet_copy_client.cc:531] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:58.095842 12062 tablet_copy_service.cc:342] P 4cd46f13bceb4c3c9ef71e755e17b995: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-41bae2134ac343ed9e9f9c1b86703d97 received from {username='slave'} at 127.0.0.1:49226
I20260808 20:13:58.096251 12062 tablet_copy_service.cc:433] P 4cd46f13bceb4c3c9ef71e755e17b995: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-41bae2134ac343ed9e9f9c1b86703d97 on tablet 41bae2134ac343ed9e9f9c1b86703d97 with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:58.103543 14093 tablet_bootstrap.cc:492] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:58.109472 13943 raft_consensus.cc:979] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9: : Instructing follower 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:13:58.110074 13942 raft_consensus.cc:1067] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Signalling peer 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:13:58.111768 12039 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:53028
I20260808 20:13:58.112252 12039 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:58.112553 12039 raft_consensus.cc:3037] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:58.123775 12039 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:58.126839 12039 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:58.127027 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:58.127501 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:58.129788 12245 raft_consensus.cc:3037] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:58.129837 12147 raft_consensus.cc:3032] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:58.130321 12147 raft_consensus.cc:729] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:58.131223 12147 consensus_queue.cc:260] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:58.132299 12147 raft_consensus.cc:3037] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:58.137488 12245 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:13:58.138567 11994 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:58.139405 14075 raft_consensus.cc:2781] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:58.139859 14075 raft_consensus.cc:686] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:58.140672 14075 consensus_queue.cc:237] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:58.148299 12147 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:13:58.150146 13800 raft_consensus.cc:979] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995: : Instructing follower b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:13:58.150624 13800 raft_consensus.cc:1067] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Signalling peer b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:13:58.152069 12245 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:44384
I20260808 20:13:58.152591 12245 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:58.152921 12245 raft_consensus.cc:3037] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:58.163668 12245 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:58.165679 12245 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:58.167304 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:58.167907 12147 raft_consensus.cc:3037] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:58.168078 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:58.168627 12039 raft_consensus.cc:3032] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:58.168916 12039 raft_consensus.cc:729] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:58.169535 12039 consensus_queue.cc:260] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:58.170770 12039 raft_consensus.cc:3037] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:58.177130 12039 raft_consensus.cc:2454] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:13:58.178501 12147 raft_consensus.cc:2454] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:13:58.184507 12206 leader_election.cc:302] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:58.183465 14093 tablet_bootstrap.cc:492] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:58.186028 13967 raft_consensus.cc:2781] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:58.186622 13967 raft_consensus.cc:686] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:58.187033 14093 tablet_bootstrap.cc:492] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:58.187475 11897 catalog_manager.cc:5495] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 1 to 2, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:58.188289 14091 ts_tablet_manager.cc:927] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:13:58.191617 13967 consensus_queue.cc:237] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:58.194201 14093 ts_tablet_manager.cc:1397] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.091s user 0.021s sys 0.021s
I20260808 20:13:58.207952 14093 raft_consensus.cc:348] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:58.208860 14093 raft_consensus.cc:729] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:58.210146 14093 consensus_queue.cc:260] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:58.215900 14091 tablet_copy_client.cc:316] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.1:40747
I20260808 20:13:58.217703 12062 tablet_copy_service.cc:140] P 4cd46f13bceb4c3c9ef71e755e17b995: Received BeginTabletCopySession request for tablet 2a41c716cfc94652a2fc6957a1bd2a78 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:49226)
I20260808 20:13:58.218354 12062 tablet_copy_service.cc:161] P 4cd46f13bceb4c3c9ef71e755e17b995: Beginning new tablet copy session on tablet 2a41c716cfc94652a2fc6957a1bd2a78 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:49226: session id = e5bd9a4616ce4acfa20dc7a002984a20-2a41c716cfc94652a2fc6957a1bd2a78
I20260808 20:13:58.221853 11897 catalog_manager.cc:5495] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 1 to 2, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 2 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:13:58.230237 14093 ts_tablet_manager.cc:1428] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.035s user 0.001s sys 0.009s
I20260808 20:13:58.232980 12061 tablet_copy_service.cc:342] P 4cd46f13bceb4c3c9ef71e755e17b995: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-0d91cb12be034dcc9ca3038df3cc85cf received from {username='slave'} at 127.0.0.1:49226
I20260808 20:13:58.233628 12061 tablet_copy_service.cc:433] P 4cd46f13bceb4c3c9ef71e755e17b995: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-0d91cb12be034dcc9ca3038df3cc85cf on tablet 0d91cb12be034dcc9ca3038df3cc85cf with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:58.240037 14093 ts_tablet_manager.cc:927] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:13:58.252267 14093 tablet_copy_client.cc:316] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.2:35139
I20260808 20:13:58.255616 12163 tablet_copy_service.cc:140] P 78b9263889cd4f1582578a3aa4f644d9: Received BeginTabletCopySession request for tablet a16032fd7a244ded8d9660c183645279 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:46680)
I20260808 20:13:58.256319 12163 tablet_copy_service.cc:161] P 78b9263889cd4f1582578a3aa4f644d9: Beginning new tablet copy session on tablet a16032fd7a244ded8d9660c183645279 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:46680: session id = e5bd9a4616ce4acfa20dc7a002984a20-a16032fd7a244ded8d9660c183645279
I20260808 20:13:58.266593 12062 tablet_copy_source_session.cc:205] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:58.267382 12163 tablet_copy_source_session.cc:205] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:13:58.269986 14091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a41c716cfc94652a2fc6957a1bd2a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:58.271638 14093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a16032fd7a244ded8d9660c183645279. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:13:58.287220 14091 tablet_copy_client.cc:791] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:58.287819 14091 tablet_copy_client.cc:655] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:58.291502 14093 tablet_copy_client.cc:791] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:13:58.291770 14091 tablet_copy_client.cc:531] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:58.292196 14093 tablet_copy_client.cc:655] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:13:58.297250 14093 tablet_copy_client.cc:531] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:13:58.309355 14091 tablet_bootstrap.cc:492] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:58.311439 14093 tablet_bootstrap.cc:492] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:13:58.353688 14093 tablet_bootstrap.cc:492] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:58.355089 14093 tablet_bootstrap.cc:492] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:58.356058 14093 ts_tablet_manager.cc:1397] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.045s user 0.015s sys 0.026s
I20260808 20:13:58.360921 14091 tablet_bootstrap.cc:492] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:13:58.361040 14093 raft_consensus.cc:348] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:58.362375 14091 tablet_bootstrap.cc:492] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:13:58.362634 14093 raft_consensus.cc:729] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:58.363361 14091 ts_tablet_manager.cc:1397] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 0.054s user 0.021s sys 0.024s
I20260808 20:13:58.363803 14093 consensus_queue.cc:260] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:58.367928 14093 ts_tablet_manager.cc:1428] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260808 20:13:58.368495 14091 raft_consensus.cc:348] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:58.369602 12163 tablet_copy_service.cc:342] P 78b9263889cd4f1582578a3aa4f644d9: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-a16032fd7a244ded8d9660c183645279 received from {username='slave'} at 127.0.0.1:46680
I20260808 20:13:58.370009 12163 tablet_copy_service.cc:433] P 78b9263889cd4f1582578a3aa4f644d9: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-a16032fd7a244ded8d9660c183645279 on tablet a16032fd7a244ded8d9660c183645279 with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:58.369338 14091 raft_consensus.cc:729] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:13:58.372471 14091 consensus_queue.cc:260] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:13:58.377385 14091 ts_tablet_manager.cc:1428] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.013s user 0.008s sys 0.002s
I20260808 20:13:58.379228 12062 tablet_copy_service.cc:342] P 4cd46f13bceb4c3c9ef71e755e17b995: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-2a41c716cfc94652a2fc6957a1bd2a78 received from {username='slave'} at 127.0.0.1:49226
I20260808 20:13:58.379604 12062 tablet_copy_service.cc:433] P 4cd46f13bceb4c3c9ef71e755e17b995: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-2a41c716cfc94652a2fc6957a1bd2a78 on tablet 2a41c716cfc94652a2fc6957a1bd2a78 with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:13:58.382983 14007 raft_consensus.cc:979] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a: : Instructing follower 78b9263889cd4f1582578a3aa4f644d9 to start an election
I20260808 20:13:58.383433 14051 raft_consensus.cc:1067] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Signalling peer 78b9263889cd4f1582578a3aa4f644d9 to start an election
I20260808 20:13:58.385377 12147 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:53162
I20260808 20:13:58.385852 12147 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:58.386178 12147 raft_consensus.cc:3037] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:58.391963 12147 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:58.393319 12147 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:58.393903 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:58.394214 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:58.394469 12045 raft_consensus.cc:3037] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:58.394739 12245 raft_consensus.cc:3032] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:58.395172 12245 raft_consensus.cc:729] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:58.395889 12245 consensus_queue.cc:260] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:58.396961 12245 raft_consensus.cc:3037] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:58.402733 12245 raft_consensus.cc:2454] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:13:58.403738 12101 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:58.413069 14084 raft_consensus.cc:2781] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:58.413524 14084 raft_consensus.cc:686] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:58.414309 14084 consensus_queue.cc:237] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:58.418002 12045 raft_consensus.cc:2454] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:13:58.436460 11897 catalog_manager.cc:5495] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 1 to 2, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 2 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:13:58.533473 13078 raft_consensus.cc:1203] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:58.545394 12140 raft_consensus.cc:1261] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:13:58.546635 13078 raft_consensus.cc:1203] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:58.547785 13944 consensus_queue.cc:1035] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:13:58.573166 12039 raft_consensus.cc:1261] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:13:58.578922 14003 consensus_queue.cc:1035] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:13:58.584331 12140 raft_consensus.cc:1261] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:58.585749 14051 consensus_queue.cc:1035] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:58.632297 12039 raft_consensus.cc:1261] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:58.637696 14007 consensus_queue.cc:1035] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:58.652580 13080 raft_consensus.cc:1203] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:13:58.694619 12245 raft_consensus.cc:1261] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:58.696614 13918 consensus_queue.cc:1035] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:58.709028 12147 raft_consensus.cc:1261] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:58.712150 13080 raft_consensus.cc:1203] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260808 20:13:58.710278 14131 consensus_queue.cc:1035] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 20:13:58.914321 12245 raft_consensus.cc:1261] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:58.915510 13942 consensus_queue.cc:1035] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:58.925083 13918 raft_consensus.cc:1050] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:58.939716 12045 raft_consensus.cc:1261] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:13:58.943037 13942 consensus_queue.cc:1035] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:13:58.927524 13918 consensus_queue.cc:237] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:58.958837 13080 raft_consensus.cc:1261] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:58.960441 12140 raft_consensus.cc:1261] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:58.960960 12244 raft_consensus.cc:1261] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:58.961804 14075 consensus_queue.cc:1035] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:58.963346 14132 consensus_queue.cc:1035] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:58.964998 14131 consensus_queue.cc:1035] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:58.982348 14132 raft_consensus.cc:2932] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:58.986609 12244 raft_consensus.cc:2932] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:58.987566 12140 raft_consensus.cc:2932] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:58.996721 13918 raft_consensus.cc:1050] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:58.998701 13918 consensus_queue.cc:237] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:59.008605 11897 catalog_manager.cc:5495] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:59.017868 12245 raft_consensus.cc:1261] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:59.018463 12140 raft_consensus.cc:1261] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:59.018790 13080 raft_consensus.cc:2932] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:59.019748 14131 consensus_queue.cc:1035] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:59.020817 14132 consensus_queue.cc:1035] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:59.022526 13078 raft_consensus.cc:1261] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:59.024060 14075 consensus_queue.cc:1035] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:13:59.031183 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:59.069252 14075 raft_consensus.cc:2932] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:59.070681 12248 raft_consensus.cc:2932] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:59.092916 12038 consensus_queue.cc:237] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:59.097937 13079 raft_consensus.cc:1261] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:59.098572 12045 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:13:59.099156 12045 raft_consensus.cc:595] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Received request to transfer leadership to TS b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:59.099361 14131 consensus_queue.cc:1035] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:59.099695 11896 catalog_manager.cc:5495] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:59.102330 12147 raft_consensus.cc:1261] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:59.104369 12147 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:49548
I20260808 20:13:59.104878 12147 raft_consensus.cc:595] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 LEADER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:13:59.105218 14131 consensus_queue.cc:1035] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:59.108202 12147 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e256c29dec964f49856cf204f950793b"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
mode: GRACEFUL
new_leader_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:49548
I20260808 20:13:59.108719 12147 raft_consensus.cc:595] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Received request to transfer leadership to TS b9d94f35ea15421aa1d721201c782d8a
I20260808 20:13:59.108011 13918 raft_consensus.cc:2932] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:59.113623 12245 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:13:59.114135 12245 raft_consensus.cc:595] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:13:59.116444 12147 raft_consensus.cc:2932] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:59.121742 13079 raft_consensus.cc:2932] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:59.123272 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260808 20:13:59.124992 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:13:59.125645 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:13:59.167968 11873 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 41bae2134ac343ed9e9f9c1b86703d97 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:59.171617 12142 raft_consensus.cc:2932] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:59.177814 13078 raft_consensus.cc:2932] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:59.191399 11889 catalog_manager.cc:5495] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:59.219939 13812 raft_consensus.cc:1050] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:59.222527 13812 consensus_queue.cc:237] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:59.231698 12045 consensus_queue.cc:237] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:59.305155 12235 tablet_service.cc:1514] Processing DeleteTablet for tablet 41bae2134ac343ed9e9f9c1b86703d97 with delete_type TABLET_DATA_TOMBSTONED (TS b9d94f35ea15421aa1d721201c782d8a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44376
I20260808 20:13:59.318748 12248 raft_consensus.cc:1261] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:59.319061 13942 raft_consensus.cc:1050] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:13:59.322081 13942 consensus_queue.cc:237] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:59.349625 13079 raft_consensus.cc:1261] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:13:59.349983 14166 tablet_replica.cc:330] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a: stopping tablet replica
I20260808 20:13:59.361548 14166 raft_consensus.cc:2229] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:59.363082 14166 raft_consensus.cc:2258] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:59.368641 14166 ts_tablet_manager.cc:1905] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260808 20:13:59.374478 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 51510.4us
I20260808 20:13:59.374604 13918 consensus_queue.cc:1035] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260808 20:13:59.375651 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 45303.3us
I20260808 20:13:59.376142 12248 raft_consensus.cc:1261] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 20:13:59.381789 13943 raft_consensus.cc:979] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9: : Instructing follower b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:13:59.382385 13943 raft_consensus.cc:1067] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Signalling peer b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:13:59.384111 12142 raft_consensus.cc:1261] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 20:13:59.386053 12244 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e256c29dec964f49856cf204f950793b"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:44378
I20260808 20:13:59.386636 12244 raft_consensus.cc:482] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:59.387063 12244 raft_consensus.cc:3037] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:59.391412 13812 consensus_queue.cc:1035] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:59.393905 14173 consensus_queue.cc:1035] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 20:13:59.396868 14131 raft_consensus.cc:2932] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:59.399284 12244 raft_consensus.cc:504] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:59.398634 12248 raft_consensus.cc:2932] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:59.404147 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:59.404662 12142 raft_consensus.cc:3032] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:59.404960 12142 raft_consensus.cc:729] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:59.409039 14084 raft_consensus.cc:979] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9: : Instructing follower 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:13:59.410442 12142 consensus_queue.cc:260] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:59.413403 12142 raft_consensus.cc:3037] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:59.417222 14084 raft_consensus.cc:1067] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 LEADER]: Signalling peer 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:13:59.420245 13079 raft_consensus.cc:1261] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEARNER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 20:13:59.424189 12040 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:53028
I20260808 20:13:59.424671 12040 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:59.425066 12040 raft_consensus.cc:3037] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:59.426692 12244 leader_election.cc:288] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:13:59.428181 12142 raft_consensus.cc:2454] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:13:59.427778 11896 catalog_manager.cc:5495] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New cstate: current_term: 1 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:59.432475 12206 leader_election.cc:302] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:59.433830 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:13:59.434376 13079 raft_consensus.cc:3037] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:59.446153 13079 raft_consensus.cc:2454] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:13:59.446184 12040 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:59.447957 12040 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:13:59.448617 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:13:59.449327 12142 raft_consensus.cc:3032] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:13:59.451007 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:13:59.451575 12244 raft_consensus.cc:3037] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:59.449596 12142 raft_consensus.cc:729] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:13:59.459208 12244 raft_consensus.cc:2454] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:13:59.454975 12142 consensus_queue.cc:260] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:59.460098 11994 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:13:59.460671 12142 raft_consensus.cc:3037] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:13:59.468792 13812 consensus_queue.cc:1035] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:13:59.472090 14176 raft_consensus.cc:2781] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:13:59.472870 14176 raft_consensus.cc:686] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:59.474926 12131 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d91cb12be034dcc9ca3038df3cc85cf with delete_type TABLET_DATA_TOMBSTONED (TS 78b9263889cd4f1582578a3aa4f644d9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53148
I20260808 20:13:59.474293 14176 consensus_queue.cc:237] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:13:59.481873 14158 raft_consensus.cc:2781] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:59.482656 14158 raft_consensus.cc:686] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:59.484412 12142 raft_consensus.cc:2454] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:13:59.490506 13078 raft_consensus.cc:1261] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:59.502998 14171 consensus_queue.cc:1035] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:13:59.510756 11873 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 0d91cb12be034dcc9ca3038df3cc85cf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:13:59.512503 13812 raft_consensus.cc:979] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995: : Instructing follower b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:13:59.512951 13812 raft_consensus.cc:1067] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Signalling peer b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:13:59.484133 14158 consensus_queue.cc:237] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:59.468001 13918 consensus_queue.cc:1035] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20260808 20:13:59.522583 12251 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:44384
I20260808 20:13:59.528726 12251 raft_consensus.cc:482] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:13:59.529192 12251 raft_consensus.cc:3037] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:59.536636 12251 raft_consensus.cc:504] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:59.538520 12251 leader_election.cc:288] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:13:59.542029 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:13:59.542649 13080 raft_consensus.cc:3037] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:59.548862 13080 raft_consensus.cc:2454] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:13:59.563642 13078 raft_consensus.cc:2932] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:13:59.579079 12244 raft_consensus.cc:1261] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:59.581280 12207 leader_election.cc:302] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:13:59.582286 14161 raft_consensus.cc:2781] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:13:59.582741 14161 raft_consensus.cc:686] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:13:59.600028 14171 consensus_queue.cc:1035] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260808 20:13:59.611778 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 182072us
I20260808 20:13:59.584188 14161 consensus_queue.cc:237] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:59.649600 11894 catalog_manager.cc:5495] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 2 to 3, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 3 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
W20260808 20:13:59.616523 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 102363us
W20260808 20:13:59.634549 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 127268us
I20260808 20:13:59.638264 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:13:59.793731 12040 raft_consensus.cc:3032] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:13:59.794131 12040 raft_consensus.cc:729] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:13:59.794970 12040 consensus_queue.cc:260] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:13:59.796212 12040 raft_consensus.cc:3037] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:13:59.836535 12040 raft_consensus.cc:2454] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:13:59.866362 14166 ts_tablet_manager.cc:1918] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:13:59.867362 14166 log.cc:1196] T 41bae2134ac343ed9e9f9c1b86703d97 P b9d94f35ea15421aa1d721201c782d8a: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-2-root/wals/41bae2134ac343ed9e9f9c1b86703d97
W20260808 20:13:59.872066 14185 log.cc:924] Time spent T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.166s user 0.000s sys 0.001s
W20260808 20:13:59.888093 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 177642us
I20260808 20:13:59.930001 14181 tablet_replica.cc:330] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9: stopping tablet replica
I20260808 20:13:59.931015 14181 raft_consensus.cc:2229] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:13:59.936792 12040 raft_consensus.cc:1261] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:13:59.932794 14181 raft_consensus.cc:2258] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:13:59.964236 14181 ts_tablet_manager.cc:1905] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:14:00.002206 14201 raft_consensus.cc:2932] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:00.006199 13078 raft_consensus.cc:2932] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:00.000974 13943 consensus_queue.cc:1035] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
W20260808 20:14:00.022624 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 375824us
I20260808 20:14:00.025339 13967 raft_consensus.cc:979] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a: : Instructing follower 78b9263889cd4f1582578a3aa4f644d9 to start an election
I20260808 20:14:00.025821 13967 raft_consensus.cc:1067] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Signalling peer 78b9263889cd4f1582578a3aa4f644d9 to start an election
I20260808 20:14:00.031298 13076 raft_consensus.cc:1261] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:13:59.986197 11872 catalog_manager.cc:4841] TS b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): tablet 41bae2134ac343ed9e9f9c1b86703d97 (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
W20260808 20:14:00.033526 11872 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 47353.6us
I20260808 20:14:00.039794 12147 raft_consensus.cc:2932] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:00.044250 14206 raft_consensus.cc:2932] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:00.045500 12039 raft_consensus.cc:2932] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:00.046775 11896 catalog_manager.cc:5495] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 1 to 2, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 2 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:00.049480 14181 ts_tablet_manager.cc:1918] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:14:00.050027 14181 log.cc:1196] T 0d91cb12be034dcc9ca3038df3cc85cf P 78b9263889cd4f1582578a3aa4f644d9: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-1-root/wals/0d91cb12be034dcc9ca3038df3cc85cf
I20260808 20:14:00.051620 11875 catalog_manager.cc:4841] TS 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): tablet 0d91cb12be034dcc9ca3038df3cc85cf (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:14:00.055521 11894 catalog_manager.cc:5495] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index 3 to 4, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:14:00.075707 12040 consensus_queue.cc:237] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:00.086669 14051 consensus_queue.cc:1035] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:00.093340 13079 raft_consensus.cc:2932] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:00.096097 11894 catalog_manager.cc:5495] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 1 to 2, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 2 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:00.104323 13076 raft_consensus.cc:1261] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:14:00.107218 12244 raft_consensus.cc:1261] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:14:00.110937 12249 raft_consensus.cc:1261] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:14:00.119073 12146 raft_consensus.cc:1261] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:14:00.115759 12248 raft_consensus.cc:2932] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:00.124637 12146 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:53162
I20260808 20:14:00.125164 12146 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:14:00.125470 12146 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:00.120361 11896 catalog_manager.cc:5495] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:00.127233 14131 consensus_queue.cc:1035] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:00.131495 12146 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:00.137449 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:00.136870 12249 raft_consensus.cc:2932] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:00.137995 12245 raft_consensus.cc:3032] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:00.138281 12245 raft_consensus.cc:729] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:00.138912 12245 consensus_queue.cc:260] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:00.139887 12245 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:00.144143 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:00.144711 12039 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:00.167647 12039 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:00.168726 12102 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:00.169425 13943 raft_consensus.cc:2781] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:00.169842 13943 raft_consensus.cc:686] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:00.170794 12245 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:00.170567 13943 consensus_queue.cc:237] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:00.184641 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:14:00.190635 14131 raft_consensus.cc:2932] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:00.194335 12147 raft_consensus.cc:1261] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:14:00.197662 14132 consensus_queue.cc:1035] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:14:00.205747 13079 raft_consensus.cc:2932] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:00.216293 13918 consensus_queue.cc:1035] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:14:00.220419 13918 consensus_queue.cc:1035] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:14:00.221515 12147 consensus_queue.cc:237] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:00.229328 11873 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 2a41c716cfc94652a2fc6957a1bd2a78 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:14:00.236523 12146 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:00.250578 13076 raft_consensus.cc:1261] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:14:00.253418 14190 consensus_queue.cc:1035] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260808 20:14:00.260421 14051 consensus_queue.cc:1035] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:00.266598 12039 raft_consensus.cc:1261] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 20:14:00.267783 13943 consensus_queue.cc:1035] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:14:00.269200 11897 catalog_manager.cc:5495] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index 4 to 5, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New cstate: current_term: 2 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260808 20:14:00.280053 14191 log.cc:924] Time spent T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.061s user 0.000s sys 0.014s
I20260808 20:14:00.279374 14188 raft_consensus.cc:2932] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
W20260808 20:14:00.281565 14135 log.cc:924] Time spent T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.063s user 0.000s sys 0.014s
I20260808 20:14:00.282262 12039 raft_consensus.cc:1261] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:14:00.285037 14051 consensus_queue.cc:1035] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:00.291303 11896 catalog_manager.cc:5495] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 2 to 3, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 3 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:00.306726 12249 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:14:00.307322 12249 raft_consensus.cc:595] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:00.308248 12245 raft_consensus.cc:2932] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:00.316032 12249 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:14:00.316627 12249 raft_consensus.cc:595] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Received request to transfer leadership to TS e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:00.323567 12039 raft_consensus.cc:2932] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:00.342705 12249 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:14:00.343361 12249 raft_consensus.cc:595] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:00.346074 12040 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:14:00.346642 12040 raft_consensus.cc:595] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:00.347820 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260808 20:14:00.352353 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:14:00.353045 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:14:00.367885 12131 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a41c716cfc94652a2fc6957a1bd2a78 with delete_type TABLET_DATA_TOMBSTONED (TS 78b9263889cd4f1582578a3aa4f644d9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:53148
I20260808 20:14:00.378226 14181 tablet_replica.cc:330] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9: stopping tablet replica
I20260808 20:14:00.379173 14181 raft_consensus.cc:2229] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 20:14:00.379865 14181 raft_consensus.cc:2258] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260808 20:14:00.454607 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 61087.1us
W20260808 20:14:00.528576 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 46335.1us
I20260808 20:14:00.534976 14181 ts_tablet_manager.cc:1905] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:14:00.548866 11875 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet a16032fd7a244ded8d9660c183645279 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 20:14:00.553613 11897 catalog_manager.cc:5495] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:00.574018 13078 raft_consensus.cc:2932] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:00.581241 14186 raft_consensus.cc:979] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a: : Instructing follower 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:14:00.581748 14186 raft_consensus.cc:1067] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Signalling peer 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:14:00.588632 13079 raft_consensus.cc:1261] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260808 20:14:00.599265 14236 log.cc:924] Time spent T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.144s user 0.002s sys 0.000s
W20260808 20:14:00.601485 14238 log.cc:924] Time spent T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.147s user 0.000s sys 0.001s
W20260808 20:14:00.637223 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 159728us
W20260808 20:14:00.641739 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 164518us
W20260808 20:14:00.642524 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 72820.5us
W20260808 20:14:00.651937 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 85329.9us
I20260808 20:14:00.660065 12235 tablet_service.cc:1514] Processing DeleteTablet for tablet a16032fd7a244ded8d9660c183645279 with delete_type TABLET_DATA_TOMBSTONED (TS b9d94f35ea15421aa1d721201c782d8a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44376
I20260808 20:14:00.661787 12038 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:53018
I20260808 20:14:00.662365 12038 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:14:00.662689 12038 raft_consensus.cc:3037] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:00.679744 13967 raft_consensus.cc:979] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a: : Instructing follower 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:14:00.680251 13967 raft_consensus.cc:1067] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Signalling peer 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:14:00.682016 12038 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:00.686161 12040 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:53018
I20260808 20:14:00.686666 12040 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:14:00.686966 12040 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:00.688863 14252 tablet_replica.cc:330] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a: stopping tablet replica
I20260808 20:14:00.684738 14201 raft_consensus.cc:979] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995: : Instructing follower 78b9263889cd4f1582578a3aa4f644d9 to start an election
I20260808 20:14:00.689846 14252 raft_consensus.cc:2229] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 20:14:00.690482 14252 raft_consensus.cc:2258] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 20:14:00.696691 14201 raft_consensus.cc:1067] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Signalling peer 78b9263889cd4f1582578a3aa4f644d9 to start an election
I20260808 20:14:00.694304 13967 consensus_queue.cc:1035] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260808 20:14:00.698266 12147 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:53176
I20260808 20:14:00.698736 12147 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:14:00.699050 12147 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:00.703738 12038 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:00.705036 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:00.706269 12152 raft_consensus.cc:3037] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:00.707424 12147 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:00.712162 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:00.713239 12038 raft_consensus.cc:3032] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:00.713590 12147 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:00.714493 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:00.715103 13079 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:00.714520 12040 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:00.717061 12249 raft_consensus.cc:1261] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:14:00.724156 12040 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:00.724309 13079 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:00.721107 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:00.727219 14181 ts_tablet_manager.cc:1918] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260808 20:14:00.718626 14252 ts_tablet_manager.cc:1905] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 20:14:00.726133 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:00.728876 12248 raft_consensus.cc:3032] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:00.729203 12248 raft_consensus.cc:729] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:00.729616 12105 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:00.729858 12248 consensus_queue.cc:260] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:00.730132 14188 consensus_queue.cc:1035] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:14:00.731092 12248 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:00.727319 12245 raft_consensus.cc:3032] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:00.732461 12245 raft_consensus.cc:729] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:00.733052 14181 log.cc:1196] T 2a41c716cfc94652a2fc6957a1bd2a78 P 78b9263889cd4f1582578a3aa4f644d9: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-1-root/wals/2a41c716cfc94652a2fc6957a1bd2a78
I20260808 20:14:00.733210 12245 consensus_queue.cc:260] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:00.734392 12245 raft_consensus.cc:3037] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:00.734642 11875 catalog_manager.cc:4841] TS 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): tablet 2a41c716cfc94652a2fc6957a1bd2a78 (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:14:00.724552 12038 raft_consensus.cc:729] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:00.739240 12038 consensus_queue.cc:260] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:00.729365 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:00.745321 12147 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:00.745733 12038 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:00.746802 14252 ts_tablet_manager.cc:1918] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 20:14:00.747283 14252 log.cc:1196] T a16032fd7a244ded8d9660c183645279 P b9d94f35ea15421aa1d721201c782d8a: Deleting WAL directory at /tmp/dist-test-taskuEkmk6/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786219941176655-10336-0/minicluster-data/ts-2-root/wals/a16032fd7a244ded8d9660c183645279
I20260808 20:14:00.748625 14188 raft_consensus.cc:2781] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:14:00.748991 11872 catalog_manager.cc:4841] TS b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): tablet a16032fd7a244ded8d9660c183645279 (table test-workload [id=b2172fa8e7574b4990306664f4c37ada]) successfully deleted
I20260808 20:14:00.749089 14188 raft_consensus.cc:686] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:00.749974 14188 consensus_queue.cc:237] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:00.753279 12248 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:00.754678 11994 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:00.755290 12040 raft_consensus.cc:1261] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:14:00.755519 14201 raft_consensus.cc:2781] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:00.755980 14201 raft_consensus.cc:686] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:00.756613 14189 consensus_queue.cc:1035] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:14:00.756811 14201 consensus_queue.cc:237] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:00.744748 12245 raft_consensus.cc:2454] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:00.758867 11994 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:00.759603 14242 raft_consensus.cc:2781] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:14:00.760082 14242 raft_consensus.cc:686] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:00.725672 12152 raft_consensus.cc:2454] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:00.761437 14242 consensus_queue.cc:237] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:00.765707 12038 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:00.770668 12147 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:00.774752 14249 raft_consensus.cc:979] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a: : Instructing follower e5bd9a4616ce4acfa20dc7a002984a20 to start an election
I20260808 20:14:00.775269 14250 raft_consensus.cc:1067] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Signalling peer e5bd9a4616ce4acfa20dc7a002984a20 to start an election
I20260808 20:14:00.781985 13079 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c"
dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
from {username='slave'} at 127.0.0.1:57330
I20260808 20:14:00.782640 13079 raft_consensus.cc:482] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:14:00.782963 13079 raft_consensus.cc:3037] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:00.791507 13079 raft_consensus.cc:504] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:00.793757 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:00.794287 12038 raft_consensus.cc:3037] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:00.795646 13079 leader_election.cc:288] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:00.797434 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:00.797952 12248 raft_consensus.cc:3032] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:00.798230 12248 raft_consensus.cc:729] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:00.798864 12248 consensus_queue.cc:260] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:00.799980 12248 raft_consensus.cc:3037] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:00.804764 12038 raft_consensus.cc:2454] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:00.810545 12248 raft_consensus.cc:2454] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:00.811652 13014 leader_election.cc:302] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:00.813753 13944 raft_consensus.cc:2781] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:00.814319 13944 raft_consensus.cc:686] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:00.815193 13944 consensus_queue.cc:237] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:00.816939 11889 catalog_manager.cc:5495] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 1 to 2, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 2 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:00.823360 11897 catalog_manager.cc:5495] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 3 to 4, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 4 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:00.827270 11897 catalog_manager.cc:5495] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 1 to 2, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:00.868803 11896 catalog_manager.cc:5495] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: term changed from 2 to 3, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4). New cstate: current_term: 3 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:14:01.142417 12147 raft_consensus.cc:1261] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:14:01.143839 14201 consensus_queue.cc:1035] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:14:01.157516 12248 raft_consensus.cc:1261] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:14:01.158514 14201 consensus_queue.cc:1035] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:14:01.202255 12038 raft_consensus.cc:1261] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:14:01.203361 14189 consensus_queue.cc:1035] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:01.210700 13079 raft_consensus.cc:1261] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:14:01.212100 14190 consensus_queue.cc:1035] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:01.344417 12251 raft_consensus.cc:1261] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 20:14:01.345435 13918 consensus_queue.cc:1035] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:14:01.354118 12147 raft_consensus.cc:1261] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 20:14:01.355314 14201 consensus_queue.cc:1035] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:14:01.413542 12038 raft_consensus.cc:1261] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 20:14:01.414527 13944 consensus_queue.cc:1035] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 20:14:01.427583 12251 raft_consensus.cc:1261] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 20:14:01.428855 14277 consensus_queue.cc:1035] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 20:14:01.560058 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:14:01.606796 12038 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:14:01.607401 12038 raft_consensus.cc:595] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Received request to transfer leadership to TS b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:01.611526 12038 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:14:01.612066 12038 raft_consensus.cc:595] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Received request to transfer leadership to TS b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:01.614295 12152 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:49548
I20260808 20:14:01.614825 12152 raft_consensus.cc:595] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:01.617082 12152 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:49548
I20260808 20:14:01.617570 12152 raft_consensus.cc:595] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:01.618472 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260808 20:14:01.619724 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:14:01.620185 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260808 20:14:01.642123 11966 auto_rebalancer.cc:254] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
W20260808 20:14:01.649619 11966 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet faebf2544dae4e21b7519aeeb8b250dc, TS 4cd46f13bceb4c3c9ef71e755e17b995 -> TS e5bd9a4616ce4acfa20dc7a002984a20 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:01.649830 11966 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet faebf2544dae4e21b7519aeeb8b250dc, TS 4cd46f13bceb4c3c9ef71e755e17b995 -> TS e5bd9a4616ce4acfa20dc7a002984a20 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:01.653287 11966 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet a83f6daa15a54982865040c0110bdda8, TS 4cd46f13bceb4c3c9ef71e755e17b995 -> TS e5bd9a4616ce4acfa20dc7a002984a20 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:01.653534 11966 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet a83f6daa15a54982865040c0110bdda8, TS 4cd46f13bceb4c3c9ef71e755e17b995 -> TS e5bd9a4616ce4acfa20dc7a002984a20 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:01.657759 11966 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6b217f6c1ea849f89667b8eb2d857206, TS 78b9263889cd4f1582578a3aa4f644d9 -> TS e5bd9a4616ce4acfa20dc7a002984a20 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:01.657974 11966 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 6b217f6c1ea849f89667b8eb2d857206, TS 78b9263889cd4f1582578a3aa4f644d9 -> TS e5bd9a4616ce4acfa20dc7a002984a20 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:01.662045 11966 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 9d6f3bbe5e92475487be203c7c89b73d, TS b9d94f35ea15421aa1d721201c782d8a -> TS e5bd9a4616ce4acfa20dc7a002984a20 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:01.662314 11966 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 9d6f3bbe5e92475487be203c7c89b73d, TS b9d94f35ea15421aa1d721201c782d8a -> TS e5bd9a4616ce4acfa20dc7a002984a20 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:01.672680 14171 raft_consensus.cc:979] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9: : Instructing follower 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:14:01.673147 14190 raft_consensus.cc:1067] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Signalling peer 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:14:01.674444 12038 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:53028
I20260808 20:14:01.674937 12038 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:14:01.675246 12038 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:01.680689 12038 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:01.682318 12038 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:01.682941 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:01.683497 12147 raft_consensus.cc:3032] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:01.683735 12147 raft_consensus.cc:729] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:01.684288 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:01.684366 12147 consensus_queue.cc:260] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:01.684789 12248 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:01.685518 12147 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:01.690667 12147 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:01.691130 12248 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:01.691612 12000 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:01.692436 14201 raft_consensus.cc:2781] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:01.692937 14201 raft_consensus.cc:686] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:01.693722 14201 consensus_queue.cc:237] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:01.703780 11889 catalog_manager.cc:5495] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 3 to 4, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 4 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:01.895721 14171 raft_consensus.cc:979] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9: : Instructing follower 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:14:01.896236 14190 raft_consensus.cc:1067] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Signalling peer 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:14:01.897511 12040 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:53028
I20260808 20:14:01.897989 12040 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:14:01.898326 12040 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:01.904485 12040 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:01.906180 12040 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:01.906803 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:01.907356 12147 raft_consensus.cc:3032] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:01.907706 12147 raft_consensus.cc:729] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:01.908293 12147 consensus_queue.cc:260] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:01.909030 12147 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:01.907128 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:01.909963 12248 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:01.913484 12147 raft_consensus.cc:2454] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:01.914315 12000 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:01.914871 12248 raft_consensus.cc:2454] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:01.914937 13918 raft_consensus.cc:2781] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:14:01.915658 13918 raft_consensus.cc:686] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:01.920302 13918 consensus_queue.cc:237] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:01.935602 11889 catalog_manager.cc:5495] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 1 to 2, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:02.005020 14242 raft_consensus.cc:979] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995: : Instructing follower b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:14:02.005415 13918 raft_consensus.cc:1067] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Signalling peer b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:14:02.006604 12251 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:44384
I20260808 20:14:02.007061 12251 raft_consensus.cc:482] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:14:02.007304 12251 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:02.012413 12251 raft_consensus.cc:504] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:02.013811 12251 leader_election.cc:288] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:02.014886 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:02.015136 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:02.015493 12147 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:02.015643 12040 raft_consensus.cc:3032] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:02.015962 12040 raft_consensus.cc:729] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:02.016652 12040 consensus_queue.cc:260] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:02.017807 12040 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:02.021670 12147 raft_consensus.cc:2454] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:02.022595 12206 leader_election.cc:302] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:02.023281 14217 raft_consensus.cc:2781] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:14:02.023710 14217 raft_consensus.cc:686] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:02.024109 12040 raft_consensus.cc:2454] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:02.024504 14217 consensus_queue.cc:237] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:02.033286 11889 catalog_manager.cc:5495] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 1 to 2, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 2 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:14:02.089277 12147 raft_consensus.cc:1261] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 20:14:02.090355 13918 consensus_queue.cc:1035] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:14:02.098960 12248 raft_consensus.cc:1261] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 20:14:02.100214 14242 consensus_queue.cc:1035] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:14:02.182063 14242 raft_consensus.cc:979] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995: : Instructing follower b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:14:02.182511 13918 raft_consensus.cc:1067] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Signalling peer b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:14:02.183619 12248 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:44384
I20260808 20:14:02.183997 12248 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:14:02.184203 12248 raft_consensus.cc:3037] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:02.188771 12248 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:02.190203 12248 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:02.191066 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:02.191627 12040 raft_consensus.cc:3032] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:02.191876 12040 raft_consensus.cc:729] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:02.192106 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:02.192673 12152 raft_consensus.cc:3037] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:02.192515 12040 consensus_queue.cc:260] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:02.193712 12040 raft_consensus.cc:3037] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:02.197516 12152 raft_consensus.cc:2454] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:02.198141 12040 raft_consensus.cc:2454] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:02.198352 12206 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:02.199039 14217 raft_consensus.cc:2781] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:02.199412 14217 raft_consensus.cc:686] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:02.200069 14217 consensus_queue.cc:237] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:02.209790 11889 catalog_manager.cc:5495] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 2 to 3, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 3 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:14:02.393184 12245 raft_consensus.cc:1261] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:14:02.395138 14242 consensus_queue.cc:1035] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 20:14:02.410642 12147 raft_consensus.cc:1261] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:14:02.415043 13918 consensus_queue.cc:1035] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 20:14:02.614454 12038 raft_consensus.cc:1261] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:14:02.615530 14303 consensus_queue.cc:1035] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:14:02.619043 12152 raft_consensus.cc:1261] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:14:02.620930 13967 consensus_queue.cc:1035] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 20:14:02.664589 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:14:02.716233 12038 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:14:02.717573 12038 raft_consensus.cc:595] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:02.721297 12251 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:14:02.721814 12251 raft_consensus.cc:595] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:02.723867 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260808 20:14:02.725581 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:14:02.726233 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:14:02.751436 12147 consensus_queue.cc:237] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:02.757850 12251 raft_consensus.cc:1261] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:14:02.759179 14190 consensus_queue.cc:1035] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260808 20:14:02.759116 12105 consensus_peers.cc:487] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2607c6f53fc243d2abb6b2f74b6e2aa6. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:14:02.763932 12038 raft_consensus.cc:1261] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:14:02.765869 14190 consensus_queue.cc:1035] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:14:02.773391 14294 raft_consensus.cc:2932] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:14:02.775055 12040 raft_consensus.cc:2932] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:14:02.784283 12248 raft_consensus.cc:2932] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:14:02.788241 12147 raft_consensus.cc:1261] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:14:02.789526 14303 consensus_queue.cc:1035] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:14:02.793251 11896 catalog_manager.cc:5495] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:14:02.798945 12040 raft_consensus.cc:1261] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:14:02.800812 14303 consensus_queue.cc:1035] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:14:02.800383 12038 consensus_queue.cc:237] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
W20260808 20:14:02.829380 12001 consensus_peers.cc:487] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eec183c460ad4f1b85ad1407c53f2eab. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:14:02.832860 12248 raft_consensus.cc:1261] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 20:14:02.833756 12152 raft_consensus.cc:1261] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 20:14:02.836473 14296 consensus_queue.cc:1035] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:14:02.840787 14299 consensus_queue.cc:1035] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:14:02.847174 13918 raft_consensus.cc:2932] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:14:02.863477 12140 raft_consensus.cc:2932] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:14:02.871434 12245 consensus_queue.cc:237] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:02.873834 11889 catalog_manager.cc:5495] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index -1 to 3, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 2 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:02.862128 12251 raft_consensus.cc:2932] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:14:02.884372 12040 raft_consensus.cc:1261] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 20:14:02.884748 12207 consensus_peers.cc:487] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb8b79bb1c59483e93af813653a73a31. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:14:02.885900 14217 consensus_queue.cc:1035] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 20:14:02.888566 12152 raft_consensus.cc:1261] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 20:14:02.889983 14217 consensus_queue.cc:1035] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:14:02.906287 13967 raft_consensus.cc:2932] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:14:02.922739 12040 raft_consensus.cc:2932] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:14:02.937326 12045 consensus_queue.cc:237] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:02.940270 11896 catalog_manager.cc:5495] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:02.951875 12140 raft_consensus.cc:2932] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
W20260808 20:14:02.977018 12001 consensus_peers.cc:487] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 -> Peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Couldn't send request to peer e5bd9a4616ce4acfa20dc7a002984a20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21889249378248049ef043f0588cebc9. This is attempt 1: this message will repeat every 5th retry.
I20260808 20:14:02.978633 14217 raft_consensus.cc:979] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a: : Instructing follower 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:14:02.978897 12251 raft_consensus.cc:1261] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 20:14:02.980620 14296 consensus_queue.cc:1035] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:14:02.980952 12147 raft_consensus.cc:1261] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 20:14:02.982398 14201 consensus_queue.cc:1035] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:14:02.979142 14217 raft_consensus.cc:1067] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Signalling peer 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:14:02.988647 12045 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:53018
I20260808 20:14:02.989217 12045 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:14:02.989552 12045 raft_consensus.cc:3037] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:02.991060 14333 raft_consensus.cc:2932] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:14:02.998225 12045 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:03.000423 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:03.000869 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:03.001137 12245 raft_consensus.cc:3032] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:03.001472 13079 raft_consensus.cc:3037] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:03.001588 12245 raft_consensus.cc:729] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:03.002535 12245 consensus_queue.cc:260] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:03.003722 12245 raft_consensus.cc:3037] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:02.995934 12140 raft_consensus.cc:2932] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:14:03.008370 13079 raft_consensus.cc:2454] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:03.009419 12001 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:03.010136 12045 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:03.010351 14296 raft_consensus.cc:2781] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:14:03.012092 14296 raft_consensus.cc:686] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:03.013059 14296 consensus_queue.cc:237] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:03.025970 12245 raft_consensus.cc:2454] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:03.038404 11896 catalog_manager.cc:5495] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 1 to 2, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:03.042064 12245 raft_consensus.cc:2932] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:14:03.043030 11889 catalog_manager.cc:5495] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: config changed from index -1 to 3, NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) added. New cstate: current_term: 2 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } } }
I20260808 20:14:03.075848 14296 raft_consensus.cc:979] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995: : Instructing follower 78b9263889cd4f1582578a3aa4f644d9 to start an election
I20260808 20:14:03.076346 14333 raft_consensus.cc:1067] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Signalling peer 78b9263889cd4f1582578a3aa4f644d9 to start an election
I20260808 20:14:03.078972 12140 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:53176
I20260808 20:14:03.079808 12140 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:14:03.080142 12140 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:03.085856 12140 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:03.088101 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:03.088622 12040 raft_consensus.cc:3032] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:03.088922 12040 raft_consensus.cc:729] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:03.089435 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:03.090015 12245 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:03.089725 12040 consensus_queue.cc:260] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:03.090793 12040 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:03.092504 12140 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:03.096994 12040 raft_consensus.cc:2454] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:03.096994 12245 raft_consensus.cc:2454] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:03.098429 12101 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:03.099165 14294 raft_consensus.cc:2781] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:03.099632 14294 raft_consensus.cc:686] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:03.100334 14294 consensus_queue.cc:237] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:03.110744 11896 catalog_manager.cc:5495] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 2 to 3, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 3 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
W20260808 20:14:03.853852 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 78429.7us
I20260808 20:14:04.202760 13944 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:04.203390 13944 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.206243 14341 ts_tablet_manager.cc:927] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:04.209064 14344 ts_tablet_manager.cc:927] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:14:04.210386 13078 thread.cc:636] tablet-copy [worker] (thread pool) Time spent creating pthread: real 0.776s user 0.006s sys 0.005s
W20260808 20:14:04.210767 13078 thread.cc:602] tablet-copy [worker] (thread pool) Time spent starting thread: real 0.777s user 0.006s sys 0.005s
W20260808 20:14:04.211166 13079 thread.cc:636] tablet-copy [worker] (thread pool) Time spent creating pthread: real 0.666s user 0.005s sys 0.003s
W20260808 20:14:04.211514 13079 thread.cc:602] tablet-copy [worker] (thread pool) Time spent starting thread: real 0.666s user 0.005s sys 0.003s
I20260808 20:14:04.349773 14296 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:04.350539 14296 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:04.352725 14296 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:04.380611 14331 raft_consensus.cc:482] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:04.381243 14331 raft_consensus.cc:504] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:04.383272 14331 leader_election.cc:288] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:04.383713 14331 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:04.384146 14331 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.386348 14331 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:04.415499 14331 raft_consensus.cc:482] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:04.416189 14331 raft_consensus.cc:504] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:04.419953 14331 leader_election.cc:288] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:04.459868 13944 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:04.564778 14296 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:04.565544 14296 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:04.584847 14296 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:04.626452 14344 tablet_copy_client.cc:316] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.2:35139
I20260808 20:14:04.641526 14341 tablet_copy_client.cc:316] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.3:38391
I20260808 20:14:04.688802 13944 raft_consensus.cc:482] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:04.689610 13944 raft_consensus.cc:504] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.691648 13944 leader_election.cc:288] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:04.692178 13944 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:04.692622 13944 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.715389 13944 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:04.719517 14349 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:04.726684 14349 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.726630 14358 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:04.719745 14348 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:04.728464 14358 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.729225 14348 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:04.731354 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:14:04.731400 14348 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:04.732362 14348 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:04.732877 14348 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:04.728816 14349 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:04.734858 14349 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:04.735291 14349 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.720109 14296 raft_consensus.cc:482] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:04.737308 14296 raft_consensus.cc:504] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:04.719512 14350 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:04.738425 14350 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.740355 14350 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:04.741292 14349 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:04.720831 14351 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:04.742023 14351 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.743113 14296 leader_election.cc:288] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:04.720863 14354 raft_consensus.cc:482] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:04.722258 14355 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:04.701292 14331 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:04.744062 14351 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:04.754863 14358 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:04.755003 14354 raft_consensus.cc:504] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:04.755781 12101 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.405s user 0.009s sys 0.004s
I20260808 20:14:04.756357 14355 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:04.757253 12101 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.406s user 0.010s sys 0.004s
I20260808 20:14:04.757171 14354 leader_election.cc:288] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:04.758306 14355 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:04.760035 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 1.40885e+06us
I20260808 20:14:04.766842 14348 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:04.767838 14354 raft_consensus.cc:482] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:04.767949 14350 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:04.768296 14363 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:04.768468 14354 raft_consensus.cc:504] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:04.768850 14363 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.770494 14354 leader_election.cc:288] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:04.770766 14363 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:04.768468 14350 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.772862 14350 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:04.755623 14331 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.776440 14331 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:04.782136 14217 raft_consensus.cc:482] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:04.782765 14217 raft_consensus.cc:504] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:04.784664 14217 leader_election.cc:288] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:04.786362 14217 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:04.786823 14217 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:04.788827 14217 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:04.789561 14358 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:04.789774 14350 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:04.790383 14350 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:04.792280 14350 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:04.792773 14350 raft_consensus.cc:482] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:04.793470 14358 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:04.794051 14350 raft_consensus.cc:504] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:04.790112 14348 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:04.793285 14354 raft_consensus.cc:482] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:04.815235 14372 raft_consensus.cc:482] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:04.830644 14372 raft_consensus.cc:504] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.834802 14370 raft_consensus.cc:482] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:04.835333 14370 raft_consensus.cc:504] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:04.836347 14375 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:04.836864 14375 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:04.837423 14370 leader_election.cc:288] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:04.834838 14217 raft_consensus.cc:482] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:04.839468 14217 raft_consensus.cc:504] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.840772 14370 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:04.841307 14370 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.841457 14217 leader_election.cc:288] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:04.843250 14370 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:04.819736 14354 raft_consensus.cc:504] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.846207 13944 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:04.846688 13944 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.828400 14358 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:04.851720 14355 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:04.853742 14355 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.856683 13944 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:04.857124 13944 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:04.845790 14354 leader_election.cc:288] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:04.863521 14375 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:04.865032 13944 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.829099 14348 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:04.871888 14372 leader_election.cc:288] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:04.873281 14171 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:04.873787 14171 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:04.875802 14171 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:04.876957 14362 raft_consensus.cc:482] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:04.877449 14362 raft_consensus.cc:504] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:04.879360 14362 leader_election.cc:288] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:04.851883 14351 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:04.881729 14351 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.868562 14350 leader_election.cc:288] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:04.908279 13944 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:04.905361 14331 raft_consensus.cc:482] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:04.902871 14362 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:04.911188 13944 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:04.911962 13944 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.869753 14358 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:04.919483 13944 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:04.921051 14372 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:04.921330 14385 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:04.921865 14385 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:04.922582 14370 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:04.922739 14384 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:04.923080 14370 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:04.923194 14384 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.924151 14331 raft_consensus.cc:504] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.924970 14361 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:04.909610 14355 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:04.926349 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:04.949410 13078 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:04.923844 14385 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:04.951982 14372 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.953598 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:04.956002 13078 raft_consensus.cc:2454] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:14:04.953769 14348 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:04.955111 14379 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:04.957209 14351 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:04.917833 14358 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:04.911569 14349 raft_consensus.cc:482] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:04.915370 14363 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:04.960536 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:04.925474 14361 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.964164 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:04.969661 12046 raft_consensus.cc:2454] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 1.
I20260808 20:14:04.960500 14385 raft_consensus.cc:482] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:04.957415 14379 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.974601 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:04.974352 14363 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:04.974669 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:04.975765 14370 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:04.961979 14349 raft_consensus.cc:504] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:04.976758 14351 raft_consensus.cc:482] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:04.977221 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:04.978288 14361 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:04.981082 14378 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:04.973803 14375 raft_consensus.cc:482] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:04.981369 14355 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:04.982528 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:04.983438 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:04.983754 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:04.984298 12207 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:04.915870 14362 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.007807 14331 leader_election.cc:288] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.009110 12036 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:14:05.009670 14362 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.011234 14372 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.000474 12046 raft_consensus.cc:2454] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 1.
I20260808 20:14:04.995281 14394 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:04.997920 14358 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:04.998275 14349 leader_election.cc:288] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:04.996825 14395 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:04.998487 14375 raft_consensus.cc:504] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.001371 14376 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:04.998991 14351 raft_consensus.cc:504] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.006649 14363 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.019565 14377 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:05.009876 14355 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.018054 14378 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.025779 14372 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:05.034292 14379 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.034425 14217 raft_consensus.cc:2781] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:05.034797 14217 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:05.035143 14217 raft_consensus.cc:3037] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:05.035702 14355 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.018368 14385 raft_consensus.cc:504] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.027760 14380 raft_consensus.cc:482] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:05.035710 14375 leader_election.cc:288] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:05.042436 12207 leader_election.cc:302] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:05.030962 14377 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.044207 14375 raft_consensus.cc:482] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:05.034803 14379 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.025279 14394 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:05.030591 14395 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.047166 14375 raft_consensus.cc:504] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.044373 14384 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.048255 14380 raft_consensus.cc:504] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.044555 14385 leader_election.cc:288] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.056047 14380 leader_election.cc:288] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.057808 14375 leader_election.cc:288] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.039224 14372 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.060238 14378 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.033318 14376 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:05.061154 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 59819.6us
I20260808 20:14:05.060595 14395 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.063269 14395 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.063381 14377 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.063687 14171 raft_consensus.cc:482] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:05.064194 14171 raft_consensus.cc:504] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.051798 14370 raft_consensus.cc:2781] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:05.065114 14370 raft_consensus.cc:482] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:05.065471 14370 raft_consensus.cc:3037] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:05.066036 14171 leader_election.cc:288] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.070916 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.071575 13079 raft_consensus.cc:2454] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:14:05.072669 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.073379 13078 raft_consensus.cc:2454] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:14:05.055187 14384 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:05.078281 14389 raft_consensus.cc:482] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:05.078358 14377 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:05.080159 14355 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:05.083945 14378 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.081202 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.081573 14171 raft_consensus.cc:482] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:05.085268 14378 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.085656 14171 raft_consensus.cc:504] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.087227 14378 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:05.088081 14378 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.087621 14171 leader_election.cc:288] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.048626 14379 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.076205 14395 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.090447 14379 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.090950 14379 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.091358 14395 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.091863 14395 raft_consensus.cc:482] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:05.091501 14379 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.092336 14395 raft_consensus.cc:504] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.088912 14378 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.088780 14372 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:05.081640 14217 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.072678 14380 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.082396 14388 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.085182 14355 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.087569 14351 leader_election.cc:288] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.098001 12152 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:49548
I20260808 20:14:05.098270 14376 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.106966 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.114979 13078 raft_consensus.cc:2454] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:14:05.113328 14303 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.118933 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.113618 12152 raft_consensus.cc:595] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:05.085093 14377 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.120834 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.116099 14303 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:05.121475 14379 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.121979 14390 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:05.113545 14372 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:05.085357 14385 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.126796 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.127467 12045 raft_consensus.cc:2454] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:05.128654 14394 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.129369 14355 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:05.129765 14394 raft_consensus.cc:482] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:05.106056 14375 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:05.130548 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.106056 14171 raft_consensus.cc:482] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:05.133291 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.122048 14376 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:05.136286 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.135205 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.137012 13015 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:05.138535 12040 raft_consensus.cc:2454] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:14:05.139227 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.139878 13944 raft_consensus.cc:2781] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:05.140256 13944 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:05.140523 13944 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:05.095193 14384 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:05.111092 14380 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.143366 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.141228 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.144187 13015 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:05.146059 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.146257 14399 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:05.147815 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.150466 14398 raft_consensus.cc:482] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.148504 12251 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:14:05.145054 14171 raft_consensus.cc:504] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.133695 14372 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.098740 14389 raft_consensus.cc:504] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.157228 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.157584 12043 raft_consensus.cc:1261] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:14:05.158322 14217 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.137303 14394 raft_consensus.cc:504] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.158731 14383 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:05.159106 14396 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.159057 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.160288 12266 tablet_copy_service.cc:140] P b9d94f35ea15421aa1d721201c782d8a: Received BeginTabletCopySession request for tablet cb8b79bb1c59483e93af813653a73a31 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:53030)
I20260808 20:14:05.160778 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.169960 12245 raft_consensus.cc:2454] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 1.
I20260808 20:14:05.161547 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.161867 14171 leader_election.cc:288] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.162547 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.136785 14385 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.177258 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.119853 14388 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.165302 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.162343 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.165854 12048 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:14:05.166054 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.179471 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.181416 12247 raft_consensus.cc:2454] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 1.
I20260808 20:14:05.182634 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.166368 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.166527 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.166868 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.141196 14376 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.186580 12251 raft_consensus.cc:2454] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 1.
I20260808 20:14:05.151010 14398 raft_consensus.cc:504] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.173954 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.174232 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.172922 12248 raft_consensus.cc:2454] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 1.
I20260808 20:14:05.138101 14375 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:05.146766 14399 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.179096 12266 tablet_copy_service.cc:161] P b9d94f35ea15421aa1d721201c782d8a: Beginning new tablet copy session on tablet cb8b79bb1c59483e93af813653a73a31 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:53030: session id = e5bd9a4616ce4acfa20dc7a002984a20-cb8b79bb1c59483e93af813653a73a31
I20260808 20:14:05.166306 14372 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:05.179423 14303 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:05.177062 14384 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:14:05.179729 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.172650 14217 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.180943 12048 raft_consensus.cc:595] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Received request to transfer leadership to TS b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:05.195384 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.169323 14355 raft_consensus.cc:2781] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:05.186383 13944 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.186260 14378 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:05.183270 14396 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:05.186535 14380 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:05.185288 14395 leader_election.cc:288] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.141510 14390 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.203421 14377 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.202898 14400 raft_consensus.cc:482] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:05.183755 14383 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.291352 14171 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.302455 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260808 20:14:05.304976 14171 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:05.290689 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 93684.6us
I20260808 20:14:05.223194 13944 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:05.309643 14389 leader_election.cc:288] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.310156 14394 leader_election.cc:288] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.294456 14395 raft_consensus.cc:979] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9: : Instructing follower 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:14:05.313509 12266 tablet_copy_source_session.cc:205] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:14:05.287014 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.306881 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260808 20:14:05.260520 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 97355.6us
I20260808 20:14:05.315904 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:14:05.317792 14375 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:05.318339 12205 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:05.318674 14385 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:05.319080 14378 raft_consensus.cc:2781] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:05.213593 14355 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:05.319199 12205 leader_election.cc:302] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:05.296553 14380 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:05.212634 14401 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.330447 12205 leader_election.cc:302] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:05.259761 14396 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:05.325342 14378 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:05.294744 14399 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.215387 14217 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.327256 14355 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
W20260808 20:14:05.338150 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 194024us
I20260808 20:14:05.345304 14398 leader_election.cc:288] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.234723 14370 raft_consensus.cc:504] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.346771 13015 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:05.355808 14411 raft_consensus.cc:1067] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Signalling peer 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:14:05.345957 14395 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.338464 14375 raft_consensus.cc:2781] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:05.333535 14383 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:05.335778 14385 raft_consensus.cc:2726] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:05.333668 14379 raft_consensus.cc:482] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:05.358476 12250 raft_consensus.cc:2454] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 1.
I20260808 20:14:05.338552 14378 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:05.336151 14394 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:05.365873 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.366852 14412 raft_consensus.cc:482] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:05.371022 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.370961 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.373497 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.374255 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.375072 14217 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.375208 14403 raft_consensus.cc:482] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:05.357270 14395 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.376453 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.329578 14380 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.357759 14375 raft_consensus.cc:482] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:05.367352 14412 raft_consensus.cc:504] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.375702 14403 raft_consensus.cc:504] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.334141 14400 raft_consensus.cc:504] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.376530 12207 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:05.473745 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:05.621852 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 250819us
I20260808 20:14:05.623099 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.448994 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.624315 14375 raft_consensus.cc:3037] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:05.361284 14351 raft_consensus.cc:2781] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260808 20:14:05.529672 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 216396us
I20260808 20:14:05.626974 12207 leader_election.cc:302] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:05.474908 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:05.511579 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 75344.4us
W20260808 20:14:05.473866 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 180245us
I20260808 20:14:05.631402 12207 leader_election.cc:302] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:05.632148 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.364032 14398 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.473289 14414 raft_consensus.cc:482] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:05.368085 14384 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.638824 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.436599 14390 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.432447 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.640882 13079 raft_consensus.cc:2454] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:05.370848 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.434831 14395 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.543392 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:05.629122 14217 raft_consensus.cc:2781] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:05.577560 14388 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:05.643316 14414 raft_consensus.cc:504] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.363456 14376 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.370568 14377 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.648109 14406 consensus_queue.cc:1035] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:14:05.641292 14385 raft_consensus.cc:2781] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:05.684192 14385 raft_consensus.cc:482] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:05.684615 14385 raft_consensus.cc:3037] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:05.685067 14377 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.686951 14377 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.359874 14401 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.688727 14414 leader_election.cc:288] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.676028 14217 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.689707 14217 raft_consensus.cc:3037] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:05.690371 14401 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.649747 14384 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.648777 14380 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.656708 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.658221 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.695398 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.695464 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.696177 12152 raft_consensus.cc:2454] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:05.658914 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.659312 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.697438 12150 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:05.697984 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.698266 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.699108 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.699704 12152 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:05.699851 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.700455 12150 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:14:05.700953 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.701586 12138 raft_consensus.cc:2454] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:05.701283 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.702230 12150 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:05.696528 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.702548 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:05.703675 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.706954 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.700507 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.660148 14378 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.661262 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.662883 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.712353 12137 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:05.543735 14411 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:05.664597 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.636529 14397 raft_consensus.cc:2781] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:05.712654 14385 raft_consensus.cc:504] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.714265 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:05.714843 12046 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:05.715848 14385 leader_election.cc:288] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.643641 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.669874 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.717698 12147 raft_consensus.cc:2454] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:05.670567 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.718490 12140 raft_consensus.cc:2454] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:14:05.474378 14171 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.630509 14351 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
W20260808 20:14:05.672827 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 316588us
I20260808 20:14:05.673871 13080 raft_consensus.cc:3037] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:05.719463 14351 raft_consensus.cc:3037] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:05.719643 14355 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.720703 13079 raft_consensus.cc:1226] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:05.713063 14378 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:05.723381 14355 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:05.713126 14411 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.363898 14379 raft_consensus.cc:504] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.726461 14379 leader_election.cc:288] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.716324 14217 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.726953 13080 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:05.382846 14394 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.679378 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.677613 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.683018 13077 raft_consensus.cc:1226] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:05.692297 14384 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.707500 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:05.709964 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 64791.1us
I20260808 20:14:05.713716 14397 raft_consensus.cc:482] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:05.643253 14398 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.728243 12250 raft_consensus.cc:2454] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:14:05.732487 14398 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.733623 12244 raft_consensus.cc:1226] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:05.727682 13074 raft_consensus.cc:1226] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:05.733866 12245 raft_consensus.cc:1226] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:05.734614 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.735210 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.735880 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.735235 12247 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:05.737035 12146 raft_consensus.cc:2454] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:05.737345 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.737617 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.738749 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.739274 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.740082 14397 raft_consensus.cc:3037] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:05.742146 12046 raft_consensus.cc:2454] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:05.740610 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.743232 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.743914 14217 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.744791 12247 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:05.731006 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:05.740686 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.748936 12205 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:05.743825 12244 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:14:05.749648 14217 raft_consensus.cc:2781] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:05.750106 14217 raft_consensus.cc:686] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:05.752266 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.753146 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.753332 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.754715 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.754895 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.758304 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.758575 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:05.760322 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 120664us
I20260808 20:14:05.761343 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.761509 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.761912 12142 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:05.762720 12141 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:05.762804 14411 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.762917 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.763526 12254 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:14:05.764214 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.764504 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.765429 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.766543 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.682497 14370 leader_election.cc:288] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.769104 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.768662 13014 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:05.770350 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.770776 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.770902 13080 raft_consensus.cc:2454] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:14:05.771775 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.772423 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.773896 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.774549 13944 raft_consensus.cc:2781] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:05.774837 13944 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.774935 13014 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:05.759788 14394 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.775271 13944 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:05.776579 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.777029 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.779358 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.779058 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.776825 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.781112 12254 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:05.781373 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.781992 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.782488 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.783063 13073 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:14:05.784039 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.787223 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.785281 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.786281 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.785866 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.786655 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.791188 14398 consensus_queue.cc:1046] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 20:14:05.791857 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.792380 14398 raft_consensus.cc:3032] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:05.792773 14398 raft_consensus.cc:729] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:05.769335 14375 raft_consensus.cc:504] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.794283 12249 raft_consensus.cc:1226] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:05.794385 14384 consensus_queue.cc:1046] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:05.794716 12105 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:05.795650 14171 raft_consensus.cc:2726] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:05.795502 14384 consensus_queue.cc:1046] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:05.784873 14351 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.795846 12105 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:05.796609 13076 raft_consensus.cc:1226] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:05.796751 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:05.797379 14351 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:05.776075 13014 leader_election.cc:302] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:05.776501 14380 raft_consensus.cc:2781] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:05.814644 14380 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.815052 14380 raft_consensus.cc:3037] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:05.815318 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.816253 12105 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:05.816645 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.817744 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.818971 14377 raft_consensus.cc:2781] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:05.819348 14377 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.819708 14377 raft_consensus.cc:3037] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:05.798116 14171 raft_consensus.cc:2726] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:05.785967 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.793519 14398 consensus_queue.cc:260] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.820802 14171 consensus_queue.cc:1046] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:05.798898 12241 raft_consensus.cc:1226] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:05.786628 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.821301 14398 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:05.795529 14379 raft_consensus.cc:3032] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:05.799440 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.799683 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.800436 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.800737 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.823622 14171 raft_consensus.cc:3032] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:05.795867 14375 leader_election.cc:288] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:05.823993 14171 raft_consensus.cc:729] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:05.824857 14171 consensus_queue.cc:260] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.826156 14171 raft_consensus.cc:3037] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:05.827967 14411 raft_consensus.cc:3032] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:05.828336 14411 raft_consensus.cc:729] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:05.829249 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.829166 14411 consensus_queue.cc:260] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.830545 14411 raft_consensus.cc:3037] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:05.830801 12101 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:05.830936 14398 consensus_queue.cc:1046] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260808 20:14:05.831925 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 55883.5us
I20260808 20:14:05.822134 14379 raft_consensus.cc:729] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:05.831558 14406 raft_consensus.cc:2781] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:05.833560 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.833936 14406 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.834313 14406 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:05.834399 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.831940 12101 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:05.834746 14422 raft_consensus.cc:2781] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:05.835165 14422 raft_consensus.cc:482] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:05.835561 14422 raft_consensus.cc:3037] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:05.836030 14390 raft_consensus.cc:2726] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:05.836266 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:05.836819 13014 leader_election.cc:302] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:05.836977 12101 leader_election.cc:302] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:05.837666 13014 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:05.837940 12101 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:05.823966 12251 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:05.802076 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.804576 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.785117 14403 leader_election.cc:288] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.805292 14412 leader_election.cc:288] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.805609 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:05.844156 12101 leader_election.cc:302] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:05.803900 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.805974 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:05.785650 14397 raft_consensus.cc:504] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.805737 14400 leader_election.cc:288] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.807036 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:05.809113 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:05.809216 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.809293 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.811972 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.812207 12254 raft_consensus.cc:2454] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:14:05.814109 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.833503 14379 consensus_queue.cc:260] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.860589 14379 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:05.861253 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.801759 13944 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.835569 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.840032 14383 raft_consensus.cc:2781] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:05.840453 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.836115 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.841008 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.841532 14409 raft_consensus.cc:2781] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:05.839784 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.841693 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.775651 14217 consensus_queue.cc:237] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:05.842106 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.843369 13014 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:05.846678 12207 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:05.849442 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:05.845662 14390 raft_consensus.cc:2781] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:05.865401 12248 raft_consensus.cc:2454] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:05.831182 14384 consensus_queue.cc:1046] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260808 20:14:05.850091 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:05.850238 14341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb8b79bb1c59483e93af813653a73a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:14:05.850872 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.851017 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:05.851325 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:05.852097 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.841666 14380 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.856329 14408 raft_consensus.cc:3032] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:05.845379 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:05.865231 12249 raft_consensus.cc:2379] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 2: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:05.867178 12245 raft_consensus.cc:1226] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:05.862280 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.875181 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.876149 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.877629 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.876432 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.869860 12207 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:05.867265 12101 leader_election.cc:302] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:05.921679 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 54471.3us
I20260808 20:14:05.868134 14400 raft_consensus.cc:2781] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:05.866853 14409 raft_consensus.cc:482] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:05.867568 14390 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.880103 14351 raft_consensus.cc:2781] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:05.865993 14383 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.880685 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:05.917421 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 54134.2us
I20260808 20:14:05.906677 14384 raft_consensus.cc:3032] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:05.922345 12101 leader_election.cc:302] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:05.875545 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:05.930810 12101 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:05.931974 12101 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:05.880303 13944 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.852851 14423 raft_consensus.cc:2726] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 20:14:05.923206 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 53408.9us
I20260808 20:14:05.906436 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.870244 14385 raft_consensus.cc:2781] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:05.875634 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.929698 13080 raft_consensus.cc:2379] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 2: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:05.929576 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.933118 14407 raft_consensus.cc:2781] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:05.933307 12207 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:05.907476 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:05.935519 12101 leader_election.cc:302] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:05.937482 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.941200 13078 raft_consensus.cc:2379] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 2: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:05.939680 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.938088 12206 leader_election.cc:302] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:05.941735 12101 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:05.924265 14400 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.924453 14408 raft_consensus.cc:729] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:05.925783 14390 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:05.926154 14409 raft_consensus.cc:3037] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:05.926780 14411 raft_consensus.cc:2726] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:05.925725 14351 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.933874 14394 raft_consensus.cc:2726] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:05.949339 12101 leader_election.cc:302] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:05.950625 14419 raft_consensus.cc:2781] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:05.951303 12207 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:05.951768 13113 debug-util.cc:398] Leaking SignalData structure 0x7b08002f9060 after lost signal to thread 10340
I20260808 20:14:05.946946 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.937094 13075 raft_consensus.cc:1261] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:14:05.936556 14412 raft_consensus.cc:2726] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:05.929950 14384 raft_consensus.cc:729] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:05.955103 14399 raft_consensus.cc:2726] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:05.929277 14383 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:05.938133 12163 tablet_copy_service.cc:140] P 78b9263889cd4f1582578a3aa4f644d9: Received BeginTabletCopySession request for tablet 2607c6f53fc243d2abb6b2f74b6e2aa6 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:46680)
I20260808 20:14:05.955880 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.957178 14351 raft_consensus.cc:3037] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:05.938437 14398 raft_consensus.cc:2781] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:05.938630 14385 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:05.969781 12163 tablet_copy_service.cc:161] P 78b9263889cd4f1582578a3aa4f644d9: Beginning new tablet copy session on tablet 2607c6f53fc243d2abb6b2f74b6e2aa6 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:46680: session id = e5bd9a4616ce4acfa20dc7a002984a20-2607c6f53fc243d2abb6b2f74b6e2aa6
I20260808 20:14:05.969985 14380 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:05.970184 14385 raft_consensus.cc:3037] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:05.962647 13073 raft_consensus.cc:1226] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:05.960048 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.947407 14423 raft_consensus.cc:2726] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:05.929522 14377 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.947956 14378 raft_consensus.cc:2781] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:05.965372 14407 raft_consensus.cc:686] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:05.954311 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:05.972924 13113 debug-util.cc:398] Leaking SignalData structure 0x7b08002f9740 after lost signal to thread 11946
I20260808 20:14:05.970029 14398 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.955889 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.965161 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.968152 14425 raft_consensus.cc:979] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995: : Instructing follower b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:14:05.959043 12101 leader_election.cc:302] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:05.962432 14415 raft_consensus.cc:2726] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:06.012283 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.980820 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:05.981321 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:05.983385 13077 raft_consensus.cc:2379] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 3: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:05.957764 14419 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.962441 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:05.954282 14400 raft_consensus.cc:3037] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:05.991349 14378 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:05.977134 14397 leader_election.cc:288] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:05.986413 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:06.011680 14426 ts_tablet_manager.cc:927] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:14:06.034129 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:06.034386 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:06.009447 14398 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
W20260808 20:14:06.038151 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 79180us
I20260808 20:14:06.034953 12146 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:06.036073 14428 ts_tablet_manager.cc:927] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:14:06.041860 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:06.021564 14415 raft_consensus.cc:2726] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.021788 14419 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:06.015643 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:06.011801 14425 raft_consensus.cc:1067] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Signalling peer b9d94f35ea15421aa1d721201c782d8a to start an election
I20260808 20:14:05.993378 14407 consensus_queue.cc:237] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:05.981999 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:05.964344 14217 raft_consensus.cc:2726] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.042779 12101 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:06.041465 14412 consensus_queue.cc:1046] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260808 20:14:06.032433 14378 raft_consensus.cc:3037] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:06.051368 13014 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:05.990163 14384 consensus_queue.cc:260] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:06.051160 12101 leader_election.cc:302] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:06.052314 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:06.096813 12249 raft_consensus.cc:2379] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 2: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
W20260808 20:14:06.096704 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 45604.1us
I20260808 20:14:06.045799 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:06.043681 13074 raft_consensus.cc:1226] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:06.098284 12101 leader_election.cc:302] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:06.089557 13072 raft_consensus.cc:2379] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 2: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:06.101778 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:06.052851 14415 raft_consensus.cc:2781] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260808 20:14:06.104959 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 191069us
I20260808 20:14:06.106343 12206 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:06.106467 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:06.097970 13113 debug-util.cc:398] Leaking SignalData structure 0x7b0800146aa0 after lost signal to thread 12083
I20260808 20:14:06.107065 14217 raft_consensus.cc:2781] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:06.107107 12146 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:06.107714 14217 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:06.107836 12206 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:06.108180 14217 raft_consensus.cc:3037] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:06.109017 13944 raft_consensus.cc:2726] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:06.109066 12206 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:06.110532 14416 raft_consensus.cc:2781] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:06.110920 14416 raft_consensus.cc:482] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:06.111263 14416 raft_consensus.cc:3037] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:06.114089 12245 raft_consensus.cc:1226] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:06.115299 12206 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:06.116428 12206 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:06.105360 14376 raft_consensus.cc:2726] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.125197 12206 leader_election.cc:302] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:06.128535 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:06.106726 14415 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:06.131451 12244 raft_consensus.cc:1226] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:06.132157 14432 raft_consensus.cc:2781] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:06.132531 14432 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:06.132864 14432 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:06.102766 14425 consensus_queue.cc:1046] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260808 20:14:06.117722 14397 raft_consensus.cc:2781] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:06.108560 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:06.119480 14399 raft_consensus.cc:2726] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.117916 14370 raft_consensus.cc:2726] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260808 20:14:06.148269 13113 debug-util.cc:398] Leaking SignalData structure 0x7b0800131ca0 after lost signal to thread 12185
W20260808 20:14:06.149645 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 218163us
I20260808 20:14:06.015177 14408 consensus_queue.cc:260] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:06.107478 14403 consensus_queue.cc:1046] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260808 20:14:06.151520 11994 leader_election.cc:302] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:06.048664 14424 consensus_queue.cc:1035] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.078s
I20260808 20:14:06.156136 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:06.152801 14377 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:06.142370 14415 raft_consensus.cc:3037] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:06.147676 14397 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:06.164441 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
W20260808 20:14:06.178745 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 133457us
I20260808 20:14:06.170002 14397 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:06.222827 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:06.234404 12147 raft_consensus.cc:2454] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:14:06.157310 14422 raft_consensus.cc:504] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:06.235993 12152 raft_consensus.cc:2454] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
W20260808 20:14:06.191779 13113 debug-util.cc:398] Leaking SignalData structure 0x7b0800354760 after lost signal to thread 12287
I20260808 20:14:06.240360 14438 raft_consensus.cc:2781] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:06.244268 14438 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:06.244689 14438 raft_consensus.cc:3037] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:06.245863 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:06.264920 14431 log.cc:924] Time spent T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
I20260808 20:14:06.264796 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:06.186801 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:06.271643 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:06.273311 14433 raft_consensus.cc:2781] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:06.273712 14433 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:06.274065 14433 raft_consensus.cc:3037] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:06.267063 14384 raft_consensus.cc:3037] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:06.273911 11896 catalog_manager.cc:5495] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 3 to 4, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 4 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:14:06.279533 14411 raft_consensus.cc:3032] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
W20260808 20:14:06.282138 13113 debug-util.cc:398] Leaking SignalData structure 0x7b0800377e60 after lost signal to thread 13118
I20260808 20:14:06.234848 14406 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:06.291124 14426 tablet_copy_client.cc:316] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.1:40747
I20260808 20:14:06.300571 12062 tablet_copy_service.cc:140] P 4cd46f13bceb4c3c9ef71e755e17b995: Received BeginTabletCopySession request for tablet eec183c460ad4f1b85ad1407c53f2eab from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:49226)
I20260808 20:14:06.301139 12062 tablet_copy_service.cc:161] P 4cd46f13bceb4c3c9ef71e755e17b995: Beginning new tablet copy session on tablet eec183c460ad4f1b85ad1407c53f2eab from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:49226: session id = e5bd9a4616ce4acfa20dc7a002984a20-eec183c460ad4f1b85ad1407c53f2eab
I20260808 20:14:06.302408 14398 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:06.286885 14428 tablet_copy_client.cc:316] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Beginning tablet copy session from remote peer at address 127.10.24.1:40747
W20260808 20:14:06.305127 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 139751us
I20260808 20:14:06.306684 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:06.291503 14411 raft_consensus.cc:729] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
W20260808 20:14:06.311272 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 186934us
I20260808 20:14:06.311962 12206 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:06.313738 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:06.305356 14436 raft_consensus.cc:3032] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:06.314329 14436 raft_consensus.cc:729] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:06.318255 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:06.318850 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:06.319154 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:06.304193 14398 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:06.312469 14217 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:06.337899 14385 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:06.338133 14436 consensus_queue.cc:260] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:06.339154 12248 raft_consensus.cc:1226] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:06.339295 14436 raft_consensus.cc:3037] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:06.341324 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:06.342464 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:06.343106 13075 raft_consensus.cc:3037] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:06.348212 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:06.321291 12206 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:06.327945 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:06.353626 14217 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:06.356079 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:06.357285 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:06.357915 14351 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:06.352470 14445 consensus_queue.cc:1046] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260808 20:14:06.359227 13078 raft_consensus.cc:1226] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:06.363194 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:06.349179 14375 raft_consensus.cc:2781] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:06.370389 14351 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:06.359753 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:06.356009 12061 tablet_copy_service.cc:140] P 4cd46f13bceb4c3c9ef71e755e17b995: Received BeginTabletCopySession request for tablet 21889249378248049ef043f0588cebc9 from peer e5bd9a4616ce4acfa20dc7a002984a20 ({username='slave'} at 127.0.0.1:49226)
I20260808 20:14:06.375926 12039 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:53028
I20260808 20:14:06.377182 12039 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:14:06.377526 12039 raft_consensus.cc:3037] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
W20260808 20:14:06.378362 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 282129us
I20260808 20:14:06.355664 14385 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:06.379524 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:06.379964 12062 tablet_copy_source_session.cc:205] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995: Tablet Copy: opened 0 blocks and 1 log segments
W20260808 20:14:06.380003 12000 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.550s (SENT)
W20260808 20:14:06.393291 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 72051.9us
I20260808 20:14:06.382815 13075 raft_consensus.cc:2454] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:06.380555 12163 tablet_copy_source_session.cc:205] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:14:06.385769 14406 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:06.385838 14412 raft_consensus.cc:3032] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:06.377590 14422 leader_election.cc:288] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:06.386631 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:06.424657 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:06.380421 14415 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:06.388638 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:06.427510 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:06.427806 12243 raft_consensus.cc:2379] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 2: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:06.376030 12061 tablet_copy_service.cc:161] P 4cd46f13bceb4c3c9ef71e755e17b995: Beginning new tablet copy session on tablet 21889249378248049ef043f0588cebc9 from peer e5bd9a4616ce4acfa20dc7a002984a20 at {username='slave'} at 127.0.0.1:49226: session id = e5bd9a4616ce4acfa20dc7a002984a20-21889249378248049ef043f0588cebc9
I20260808 20:14:06.428666 12207 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:06.380532 14378 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:06.392701 14390 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:06.379240 14217 raft_consensus.cc:2781] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:06.388271 13014 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:06.433123 14217 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:06.374092 14375 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:06.420462 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:06.420969 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:06.424642 14412 raft_consensus.cc:729] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:06.425144 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:06.426229 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:06.427644 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:06.389362 13016 leader_election.cc:302] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:06.447122 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 94311.3us
I20260808 20:14:06.448253 13016 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:06.313503 14432 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
W20260808 20:14:06.432739 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 121922us
I20260808 20:14:06.433470 14217 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:06.433620 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:14:06.434530 12150 raft_consensus.cc:2454] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:14:06.420462 13076 raft_consensus.cc:1226] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260808 20:14:06.421279 12036 raft_consensus.cc:1226] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260808 20:14:06.425971 14341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:57330 (request call id 239) took 2987 ms. Trace:
W20260808 20:14:06.444640 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 56442.2us
I20260808 20:14:06.467770 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:06.468191 13014 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:06.469026 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:06.445912 14446 raft_consensus.cc:2781] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:06.387441 14383 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:06.449358 13016 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:06.449834 14389 raft_consensus.cc:2741] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260808 20:14:06.474503 14383 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:06.475030 13079 raft_consensus.cc:1226] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:06.476071 12102 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:06.477367 13016 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:06.477504 12102 leader_election.cc:302] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:06.478493 12102 leader_election.cc:302] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:06.479471 12102 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:06.481683 14420 raft_consensus.cc:2726] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:06.483044 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:06.449996 12206 leader_election.cc:302] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:06.451669 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:06.457868 14449 raft_consensus.cc:2726] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.459484 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:06.435837 14375 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:06.362753 14411 consensus_queue.cc:260] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:06.546702 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:06.462728 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:06.489694 14412 consensus_queue.cc:260] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:06.566211 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 213226us
W20260808 20:14:06.527014 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 49717.9us
I20260808 20:14:06.567766 13016 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:06.568960 13016 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:06.570009 13016 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:06.451924 14422 raft_consensus.cc:2726] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.529433 12102 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:06.484611 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:06.549618 12043 raft_consensus.cc:1261] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 20:14:06.551322 13014 leader_election.cc:302] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:06.534747 12152 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:06.546993 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:06.532191 14217 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
W20260808 20:14:06.557703 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 107785us
I20260808 20:14:06.552125 12039 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:06.434724 14397 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:06.571516 13944 raft_consensus.cc:2726] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.573783 13016 leader_election.cc:302] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:06.573388 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 44036.4us
I20260808 20:14:06.526695 14383 raft_consensus.cc:2726] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.527842 14351 raft_consensus.cc:2726] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.527647 14384 raft_consensus.cc:2726] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.582083 12039 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:06.571409 14446 raft_consensus.cc:686] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:06.527055 14445 raft_consensus.cc:2726] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:06.527109 14394 raft_consensus.cc:2726] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.578936 12206 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:06.576128 14408 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:06.587205 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:06.458755 14419 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:06.579978 12102 leader_election.cc:302] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:06.590428 13016 leader_election.cc:302] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:06.591077 13944 raft_consensus.cc:2726] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.529753 14376 raft_consensus.cc:3032] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:06.586549 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:06.547230 14432 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:06.569191 14399 raft_consensus.cc:2726] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.572325 12243 raft_consensus.cc:1226] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:06.610425 14407 raft_consensus.cc:2781] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:06.597730 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:06.611685 14390 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:06.576349 14355 raft_consensus.cc:2726] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.612401 14390 raft_consensus.cc:2726] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:06.612974 14376 raft_consensus.cc:729] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
W20260808 20:14:06.614025 14416 consensus_meta.cc:287] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.502s user 0.009s sys 0.005s
I20260808 20:14:06.614403 14376 consensus_queue.cc:260] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:06.591668 14415 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:06.528383 14171 raft_consensus.cc:2726] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:06.614655 14416 raft_consensus.cc:504] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:06.601336 14443 consensus_queue.cc:1046] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260808 20:14:06.602285 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:06.579322 14217 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:06.604686 12206 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:06.625404 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:06.577798 14424 consensus_queue.cc:1046] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260808 20:14:06.606506 14402 raft_consensus.cc:2726] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.580291 14380 raft_consensus.cc:2726] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.609620 13016 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:06.609808 14414 raft_consensus.cc:2726] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.610980 14407 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:06.598316 12036 raft_consensus.cc:1226] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:06.594089 14411 raft_consensus.cc:3037] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:06.609616 12102 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:06.622293 14419 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:06.626376 12206 leader_election.cc:302] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:06.579077 14395 raft_consensus.cc:2726] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:06.601655 14430 consensus_queue.cc:1046] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260808 20:14:06.663224 14217 raft_consensus.cc:2781] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:06.663321 12000 leader_election.cc:302] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:06.663607 14217 raft_consensus.cc:482] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:06.664005 14217 raft_consensus.cc:3037] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:06.634608 14397 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:06.619679 14376 raft_consensus.cc:3037] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:06.662158 14398 raft_consensus.cc:2726] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:06.639834 14407 raft_consensus.cc:3037] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
W20260808 20:14:06.661666 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 52133.9us
I20260808 20:14:06.682561 12102 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:06.598459 14452 consensus_queue.cc:1035] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260808 20:14:06.637632 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:06.630396 14388 raft_consensus.cc:2781] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:06.661782 12206 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:06.625331 14445 consensus_queue.cc:1046] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260808 20:14:06.685932 13079 raft_consensus.cc:2379] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 2: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:06.688382 12102 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:06.638669 14355 raft_consensus.cc:2726] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.638875 14378 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:06.696496 12102 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:06.694321 12206 leader_election.cc:302] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:06.697917 12206 leader_election.cc:302] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:06.698735 12206 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:06.686930 14388 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:06.699501 14388 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:06.702229 12206 leader_election.cc:302] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:06.689900 14455 log.cc:924] Time spent T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.058s user 0.002s sys 0.000s
I20260808 20:14:06.704677 12102 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:06.705942 14413 raft_consensus.cc:2726] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.706126 14416 leader_election.cc:288] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:06.707726 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:06.694140 14397 raft_consensus.cc:2781] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:06.669181 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:06.709059 11994 leader_election.cc:302] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:06.712672 12102 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:06.718966 14435 raft_consensus.cc:2726] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.693187 14171 raft_consensus.cc:2726] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:06.719904 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:06.720082 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:06.702713 14395 raft_consensus.cc:2726] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.727758 12102 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:06.727239 14416 raft_consensus.cc:2726] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:06.734302 12248 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:44384
I20260808 20:14:06.736073 12244 raft_consensus.cc:1226] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:06.737862 12245 raft_consensus.cc:1226] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:06.738302 12254 raft_consensus.cc:1226] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:06.710999 14452 raft_consensus.cc:2726] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.634433 14446 consensus_queue.cc:237] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:06.713411 14394 raft_consensus.cc:2726] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:06.722522 14397 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:06.757936 12248 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:14:06.758665 12205 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:06.700971 14433 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:06.712713 14398 raft_consensus.cc:2726] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.759397 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:06.761262 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:06.762615 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:06.767539 14375 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:06.685539 14454 consensus_queue.cc:1046] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.187s
I20260808 20:14:06.470248 14341 rpcz_store.cc:276] 0808 20:14:03.430289 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:03.433524 (+ 3235us) service_pool.cc:227] Handling call
0808 20:14:04.206631 (+773107us) ts_tablet_manager.cc:928] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
0808 20:14:05.844385 (+1637754us) tablet_copy_client.cc:361] Tablet copy session begun
0808 20:14:06.150003 (+305618us) tablet_metadata.cc:641] Metadata flushed
0808 20:14:06.166166 (+ 16163us) tablet_copy_client.cc:490] Wrote new tablet metadata
0808 20:14:06.384286 (+218120us) tablet_copy_client.cc:411] Wrote new consensus metadata
0808 20:14:06.384564 (+ 278us) tablet_replica.cc:205] Creating consensus instance
0808 20:14:06.384886 (+ 322us) ts_tablet_manager.cc:983] Replica registered.
0808 20:14:06.417873 (+ 32987us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":248,"tablet-copy.queue_time_us":772062,"thread_start_us":1210926,"threads_started":2}
I20260808 20:14:06.757463 12102 leader_election.cc:302] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:06.758989 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:06.722002 14378 raft_consensus.cc:2726] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.792111 12248 raft_consensus.cc:3037] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:06.755813 14445 raft_consensus.cc:2726] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:06.792665 14397 raft_consensus.cc:3037] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:14:06.802197 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 43585.7us
W20260808 20:14:06.802302 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 44921.2us
I20260808 20:14:06.817945 12102 leader_election.cc:302] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:06.819171 12102 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:06.819542 14409 consensus_meta.cc:287] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.860s user 0.007s sys 0.015s
I20260808 20:14:06.819892 14409 raft_consensus.cc:504] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:06.820549 14394 raft_consensus.cc:2726] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:06.821491 14394 raft_consensus.cc:2726] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:06.809593 14435 raft_consensus.cc:2726] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.804764 12061 tablet_copy_source_session.cc:205] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 20:14:06.823272 12102 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:06.824076 14445 raft_consensus.cc:2726] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:06.804214 14411 raft_consensus.cc:2726] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:06.824558 14388 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:06.826135 14438 consensus_meta.cc:287] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.581s user 0.009s sys 0.002s
I20260808 20:14:06.826521 14438 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:06.830345 14400 consensus_meta.cc:287] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.809s user 0.000s sys 0.018s
I20260808 20:14:06.830721 14400 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:06.831687 13016 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:06.833119 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:06.833364 14389 raft_consensus.cc:2781] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:06.833873 14389 raft_consensus.cc:686] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:06.812392 12248 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:06.834730 14389 consensus_queue.cc:237] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:06.801785 14460 raft_consensus.cc:2726] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.836577 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:06.838573 12038 raft_consensus.cc:1226] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:06.835155 14412 raft_consensus.cc:3037] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:06.817080 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:06.828655 14438 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:06.840798 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:06.803601 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:06.844058 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:06.844538 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:06.844848 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:06.803759 14344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2607c6f53fc243d2abb6b2f74b6e2aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:14:06.846131 14217 raft_consensus.cc:504] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:06.848531 12102 leader_election.cc:302] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:06.850692 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 141727us
I20260808 20:14:06.859301 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:06.860599 14430 raft_consensus.cc:2726] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:06.861264 14430 raft_consensus.cc:3032] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:06.864122 14217 leader_election.cc:288] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:06.859167 14445 raft_consensus.cc:2726] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.866209 14433 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:06.866679 12137 raft_consensus.cc:1261] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260808 20:14:06.867122 12137 pending_rounds.cc:76] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9: Aborting all ops after (but not including) 2
I20260808 20:14:06.867363 12137 pending_rounds.cc:97] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260808 20:14:06.803942 14375 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:06.803625 14341 tablet_copy_client.cc:791] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:14:06.843050 14400 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:06.859778 12102 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:06.869767 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:06.872979 12137 raft_consensus.cc:2866] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:06.869552 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:06.858341 14411 consensus_queue.cc:1046] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260808 20:14:06.848419 14388 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:06.890246 14445 raft_consensus.cc:3032] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:06.890669 14445 raft_consensus.cc:729] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:06.893491 12102 leader_election.cc:302] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:06.891985 14445 consensus_queue.cc:260] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:06.906446 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:06.908179 12102 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:06.909750 12038 raft_consensus.cc:3032] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:06.910069 12038 raft_consensus.cc:729] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:06.910770 12038 consensus_queue.cc:260] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:06.911708 12038 raft_consensus.cc:3037] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 4
I20260808 20:14:06.915598 12248 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:06.899356 14465 consensus_queue.cc:1046] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:06.918802 14398 raft_consensus.cc:2726] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:06.919463 14407 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:06.861755 14430 raft_consensus.cc:729] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:06.889633 14394 raft_consensus.cc:2726] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:14:06.932185 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:06.914453 14408 raft_consensus.cc:2726] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:06.948355 13071 raft_consensus.cc:1226] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:06.949218 14450 raft_consensus.cc:3032] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:06.949815 14450 raft_consensus.cc:729] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:06.958068 12038 raft_consensus.cc:2454] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:06.950976 14450 consensus_queue.cc:260] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:06.959197 12205 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:06.959892 14450 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:06.960100 14435 raft_consensus.cc:2781] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:06.961071 14435 raft_consensus.cc:686] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:06.921211 14341 tablet_copy_client.cc:655] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:14:06.970815 14430 consensus_queue.cc:260] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:06.974524 14430 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:06.978475 14460 raft_consensus.cc:3032] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:06.979031 14460 raft_consensus.cc:729] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:06.996037 14407 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:06.996517 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:06.970259 14409 leader_election.cc:288] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:06.923525 14436 consensus_queue.cc:1046] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
I20260808 20:14:07.001134 14443 raft_consensus.cc:3032] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:06.916309 14438 consensus_queue.cc:1046] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
W20260808 20:14:07.003979 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 336934us
I20260808 20:14:06.989583 14445 raft_consensus.cc:3037] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:06.963212 14400 consensus_queue.cc:1046] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
W20260808 20:14:07.004784 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 207330us
I20260808 20:14:07.001645 14443 raft_consensus.cc:729] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
W20260808 20:14:07.007803 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 430782us
I20260808 20:14:07.008466 12206 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:07.009763 13014 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:07.010641 12000 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:07.011162 13014 leader_election.cc:302] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:07.011893 12206 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:07.013479 12206 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:07.013682 13014 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:07.017306 12206 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:07.022562 14426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eec183c460ad4f1b85ad1407c53f2eab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:14:07.029474 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:07.031487 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:07.032270 12248 raft_consensus.cc:2379] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 2: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:06.990172 14435 consensus_queue.cc:237] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:07.045396 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:07.047065 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:07.048453 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:07.051223 12206 leader_election.cc:302] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:07.051937 14217 raft_consensus.cc:2726] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:07.053321 12000 leader_election.cc:302] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:07.014959 14396 raft_consensus.cc:2726] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:14:07.016238 14402 raft_consensus.cc:2726] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:07.014791 14383 raft_consensus.cc:2726] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:07.042186 13076 raft_consensus.cc:1226] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:07.055841 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:07.017457 14351 raft_consensus.cc:2726] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:07.046499 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:07.021461 14388 raft_consensus.cc:2726] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:07.059684 12000 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:07.056672 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:07.077108 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:07.056172 14443 consensus_queue.cc:260] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:07.062645 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:06.980576 14460 consensus_queue.cc:260] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:07.091064 14460 raft_consensus.cc:3037] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:07.019805 14465 raft_consensus.cc:2726] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:07.063944 14397 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:07.094262 12251 raft_consensus.cc:1226] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:07.095618 14397 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:07.025049 14375 raft_consensus.cc:2726] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:07.096453 12247 raft_consensus.cc:1226] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:07.073148 14341 tablet_copy_client.cc:531] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:14:07.081790 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:07.034993 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:07.067730 14436 raft_consensus.cc:2726] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:07.082401 12206 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:07.076637 12142 raft_consensus.cc:1261] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 20:14:07.077167 11994 leader_election.cc:302] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:07.061733 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:07.104636 12146 raft_consensus.cc:3037] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 4
I20260808 20:14:07.077723 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:07.093984 14424 raft_consensus.cc:2726] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:07.096685 12137 raft_consensus.cc:2454] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:14:07.077656 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:07.077231 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:07.102291 12000 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:07.047912 14433 consensus_queue.cc:1035] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20260808 20:14:07.107064 13071 raft_consensus.cc:3037] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:07.109215 14375 raft_consensus.cc:2726] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:07.111466 14408 raft_consensus.cc:3032] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:07.114775 12206 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:07.077883 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:07.118189 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 50278.6us
I20260808 20:14:07.120870 12000 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:07.127349 12000 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:07.128453 12000 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:07.130522 12000 leader_election.cc:302] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:07.130977 11994 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:07.131155 12206 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:07.124752 13075 raft_consensus.cc:1226] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:07.132581 14413 raft_consensus.cc:2726] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:14:07.132956 14457 raft_consensus.cc:3032] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:07.133576 14457 raft_consensus.cc:729] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:07.135540 13016 leader_election.cc:302] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:07.135983 12206 leader_election.cc:302] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:07.137110 14397 raft_consensus.cc:2726] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:07.137048 14409 raft_consensus.cc:2781] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:14:07.138917 14409 raft_consensus.cc:686] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:07.141614 14428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21889249378248049ef043f0588cebc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 20:14:07.124876 12243 raft_consensus.cc:2379] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 2: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:07.121145 14467 raft_consensus.cc:3032] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:07.123231 14436 raft_consensus.cc:2726] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:07.126477 14443 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:07.123476 14408 raft_consensus.cc:729] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:07.129338 14424 raft_consensus.cc:2726] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:07.143244 14454 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:07.164140 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:07.168628 14438 raft_consensus.cc:2781] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:07.173081 12146 raft_consensus.cc:2454] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:07.177127 14217 consensus_queue.cc:1035] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:07.155509 14467 raft_consensus.cc:729] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:07.140419 14409 consensus_queue.cc:237] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:07.180127 12000 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:07.179188 14407 raft_consensus.cc:2726] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:07.184540 12241 raft_consensus.cc:1226] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:07.173070 14465 raft_consensus.cc:2726] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:07.242764 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:07.171515 14436 raft_consensus.cc:2726] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:07.172273 14438 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:07.246930 11889 catalog_manager.cc:5495] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: term changed from 2 to 3, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4). New cstate: current_term: 3 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260808 20:14:07.250820 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 53311.2us
I20260808 20:14:07.173403 14469 raft_consensus.cc:2781] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260808 20:14:07.248337 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 68298.6us
W20260808 20:14:07.248780 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 59041.9us
I20260808 20:14:07.253036 14438 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:07.254891 12000 leader_election.cc:302] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:07.306952 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 52112.7us
I20260808 20:14:07.312520 12000 leader_election.cc:302] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:07.319025 12000 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:07.320206 12000 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:07.321136 14465 raft_consensus.cc:2726] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:07.322598 13079 raft_consensus.cc:1261] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 20:14:07.253239 14450 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:07.331075 12000 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:07.333901 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:07.335783 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:07.336833 12138 raft_consensus.cc:2454] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:07.339943 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:07.340248 14433 consensus_queue.cc:1035] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:07.346419 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:07.344650 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:07.344942 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:07.253464 14469 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:07.349692 14469 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:07.351369 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:07.352026 12152 raft_consensus.cc:2454] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:07.354058 12140 raft_consensus.cc:1226] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:07.350634 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:07.330066 14424 raft_consensus.cc:2726] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:07.358839 13079 raft_consensus.cc:1261] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 20:14:07.174130 14457 consensus_queue.cc:260] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:07.319670 14436 raft_consensus.cc:2726] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:07.356519 12000 leader_election.cc:302] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:07.362561 12152 raft_consensus.cc:1226] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:07.347630 14430 raft_consensus.cc:2726] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:07.369390 12000 leader_election.cc:302] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:07.370093 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:07.255695 14468 consensus_queue.cc:1046] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.103s
I20260808 20:14:07.335834 14474 raft_consensus.cc:2726] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:07.385079 14468 raft_consensus.cc:3032] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:07.385540 14468 raft_consensus.cc:729] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:07.389775 12000 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:07.390937 12000 leader_election.cc:302] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:07.376264 14472 consensus_queue.cc:1035] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260808 20:14:07.372848 14465 raft_consensus.cc:2781] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:07.381495 14388 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
W20260808 20:14:07.407588 14473 log.cc:924] Time spent T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.090s user 0.000s sys 0.002s
W20260808 20:14:07.411527 14476 log.cc:924] Time spent T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
I20260808 20:14:07.395066 14436 raft_consensus.cc:2781] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:07.392994 14430 raft_consensus.cc:2726] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:07.472532 14341 tablet_bootstrap.cc:492] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:14:07.444567 14462 consensus_queue.cc:1046] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:07.401238 14465 raft_consensus.cc:482] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:07.362124 14467 consensus_queue.cc:260] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:07.396908 14424 raft_consensus.cc:2726] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:07.442231 14468 consensus_queue.cc:260] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:07.460865 14436 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
W20260808 20:14:07.474213 14412 consensus_meta.cc:287] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.627s user 0.009s sys 0.003s
I20260808 20:14:07.474556 13071 raft_consensus.cc:2454] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:07.445714 12000 leader_election.cc:302] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:07.516029 14457 raft_consensus.cc:3037] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
W20260808 20:14:07.516674 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 71055.6us
I20260808 20:14:07.387151 14438 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:07.519539 14438 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:07.417354 14474 consensus_queue.cc:1046] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
I20260808 20:14:07.520886 12140 raft_consensus.cc:1226] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:07.460206 12047 raft_consensus.cc:1226] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:07.472404 12137 raft_consensus.cc:2454] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:07.490108 12144 raft_consensus.cc:1261] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 20:14:07.490674 14469 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:07.524001 14461 log.cc:924] Time spent T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045: Append to log took a long time: real 0.150s user 0.002s sys 0.000s
I20260808 20:14:07.524106 14430 raft_consensus.cc:2726] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 20:14:07.499106 14351 consensus_queue.cc:1159] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "78b9263889cd4f1582578a3aa4f644d9" responder_term: 4 status { last_received { term: 4 index: 7 } last_committed_idx: 7 last_received_current_leader { term: 4 index: 7 } }
I20260808 20:14:07.450420 14388 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:07.503599 14467 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:07.460244 14454 raft_consensus.cc:2726] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:07.478966 14465 raft_consensus.cc:3037] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:07.509886 14462 raft_consensus.cc:3032] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:07.510543 14436 raft_consensus.cc:3037] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:07.499133 14468 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:07.529654 14474 raft_consensus.cc:3032] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:07.532145 14422 consensus_queue.cc:1035] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 20:14:07.538108 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:07.538589 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:07.538125 12140 raft_consensus.cc:1226] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:07.541983 12138 raft_consensus.cc:1226] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:07.496161 14408 consensus_queue.cc:260] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:07.517719 14450 raft_consensus.cc:2726] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:07.560674 12000 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:07.561251 12147 raft_consensus.cc:2454] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:07.558701 14462 raft_consensus.cc:729] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:07.559727 14474 raft_consensus.cc:729] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:07.567698 14388 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:07.579212 12206 leader_election.cc:302] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:07.573787 14443 raft_consensus.cc:2781] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:14:07.584879 14469 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:07.585098 14450 consensus_queue.cc:1046] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:07.569255 14454 consensus_queue.cc:1046] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260808 20:14:07.575605 14412 consensus_queue.cc:1046] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260808 20:14:07.585502 14443 raft_consensus.cc:686] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:07.585624 14446 raft_consensus.cc:2781] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:14:07.574884 14424 consensus_queue.cc:1046] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
W20260808 20:14:07.602990 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 472060us
W20260808 20:14:07.610811 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 246539us
W20260808 20:14:07.596161 14435 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.011s sys 0.047s
W20260808 20:14:07.620050 14435 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.528s user 0.011s sys 0.047s
I20260808 20:14:07.611320 14446 raft_consensus.cc:686] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:07.631438 12254 raft_consensus.cc:1226] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:07.624913 14444 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:07.633828 14408 raft_consensus.cc:3037] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:07.637667 12047 raft_consensus.cc:1226] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:07.637877 14469 raft_consensus.cc:3032] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:07.638373 12241 raft_consensus.cc:1226] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:07.641091 14452 raft_consensus.cc:2726] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:07.643323 12146 raft_consensus.cc:1226] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:07.644588 12250 raft_consensus.cc:1226] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:07.645803 11994 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:07.654762 12249 raft_consensus.cc:1226] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:07.594803 14462 consensus_queue.cc:260] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:07.626935 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:07.670744 13071 raft_consensus.cc:2379] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 3: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:07.657594 14443 consensus_queue.cc:237] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:07.659684 14344 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:53296 (request call id 522) took 4297 ms. Trace:
I20260808 20:14:07.660799 12038 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:14:07.609287 14474 consensus_queue.cc:260] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:07.668298 12039 raft_consensus.cc:1261] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 20:14:07.656023 12247 raft_consensus.cc:1226] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:07.655939 14469 raft_consensus.cc:729] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:07.686793 11994 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:07.653195 14444 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:07.702234 14469 consensus_queue.cc:260] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:07.740823 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 52500.1us
I20260808 20:14:07.740291 14469 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:07.750928 14462 raft_consensus.cc:3037] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:07.752310 14474 raft_consensus.cc:3037] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:07.754743 14465 raft_consensus.cc:504] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:07.731918 14454 raft_consensus.cc:3032] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:07.679246 14478 raft_consensus.cc:2726] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:07.689035 14460 raft_consensus.cc:2726] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:07.685881 14425 raft_consensus.cc:3032] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:07.733508 12142 raft_consensus.cc:2454] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:07.669140 14467 consensus_queue.cc:1046] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20260808 20:14:07.735949 14426 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:53290 (request call id 398) took 1858 ms. Trace:
I20260808 20:14:07.678332 14446 consensus_queue.cc:237] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:07.739094 14444 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:07.741724 11994 leader_election.cc:302] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:07.763497 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:07.780675 14425 raft_consensus.cc:729] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:07.757129 14413 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:07.782668 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:07.783519 12205 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:07.748385 14436 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:07.785135 12150 raft_consensus.cc:2379] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 5: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
W20260808 20:14:07.789346 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 47696.7us
I20260808 20:14:07.797629 11994 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:07.798812 11994 leader_election.cc:302] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:07.784266 14478 raft_consensus.cc:3032] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:07.801577 14478 raft_consensus.cc:729] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:07.760141 14454 raft_consensus.cc:729] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:07.802510 14460 raft_consensus.cc:2726] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:07.802510 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:07.805032 11994 leader_election.cc:302] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:07.778136 12047 raft_consensus.cc:1226] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:07.780437 14426 rpcz_store.cc:276] 0808 20:14:05.877620 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:05.877750 (+ 130us) service_pool.cc:227] Handling call
0808 20:14:06.037726 (+159976us) ts_tablet_manager.cc:928] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
0808 20:14:06.556266 (+518540us) tablet_copy_client.cc:361] Tablet copy session begun
0808 20:14:07.414227 (+857961us) tablet_metadata.cc:641] Metadata flushed
0808 20:14:07.414466 (+ 239us) tablet_copy_client.cc:490] Wrote new tablet metadata
0808 20:14:07.735140 (+320674us) tablet_copy_client.cc:411] Wrote new consensus metadata
0808 20:14:07.735503 (+ 363us) tablet_replica.cc:205] Creating consensus instance
0808 20:14:07.735823 (+ 320us) ts_tablet_manager.cc:983] Replica registered.
0808 20:14:07.735874 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":21008,"tablet-copy.queue_time_us":78123,"thread_start_us":272425,"threads_started":2}
I20260808 20:14:07.679455 14344 rpcz_store.cc:276] 0808 20:14:03.362391 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:03.545016 (+182625us) service_pool.cc:227] Handling call
0808 20:14:04.209369 (+664353us) ts_tablet_manager.cc:928] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
0808 20:14:06.659619 (+2450250us) tablet_copy_client.cc:361] Tablet copy session begun
0808 20:14:07.250873 (+591254us) tablet_metadata.cc:641] Metadata flushed
0808 20:14:07.318391 (+ 67518us) tablet_copy_client.cc:490] Wrote new tablet metadata
0808 20:14:07.640861 (+322470us) tablet_copy_client.cc:411] Wrote new consensus metadata
0808 20:14:07.659049 (+ 18188us) tablet_replica.cc:205] Creating consensus instance
0808 20:14:07.659498 (+ 449us) ts_tablet_manager.cc:983] Replica registered.
0808 20:14:07.659605 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":242,"tablet-copy.queue_time_us":663491,"thread_start_us":1082593,"threads_started":2}
I20260808 20:14:07.806226 14460 raft_consensus.cc:2726] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:07.803115 14454 consensus_queue.cc:260] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:07.811156 14344 tablet_copy_client.cc:791] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:14:07.758337 14465 leader_election.cc:288] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:07.826395 12150 raft_consensus.cc:1261] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:14:07.828714 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:07.829394 12045 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:07.804253 14467 raft_consensus.cc:2726] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:07.811847 14426 tablet_copy_client.cc:791] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
I20260808 20:14:07.833223 12147 raft_consensus.cc:2454] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:07.807498 14444 raft_consensus.cc:2726] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 20:14:07.837317 12000 leader_election.cc:302] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:07.802416 14478 consensus_queue.cc:260] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:07.834719 14344 tablet_copy_client.cc:655] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:14:07.840639 14478 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:07.834821 14465 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:07.845348 12205 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:07.821743 14430 raft_consensus.cc:2726] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:07.846318 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:07.846294 12001 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:07.826884 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:07.807381 14413 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:07.858989 14396 raft_consensus.cc:2781] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:07.859035 12041 raft_consensus.cc:1261] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260808 20:14:07.859532 12041 pending_rounds.cc:76] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting all ops after (but not including) 4
I20260808 20:14:07.859817 12041 pending_rounds.cc:97] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260808 20:14:07.860049 12041 raft_consensus.cc:2866] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:07.847589 14425 consensus_queue.cc:260] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:07.847494 14422 consensus_queue.cc:1035] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260808 20:14:07.847549 14426 tablet_copy_client.cc:655] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:14:07.859438 14396 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:07.831750 14450 consensus_queue.cc:1046] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20260808 20:14:07.846751 14457 raft_consensus.cc:2781] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:14:07.850910 14454 raft_consensus.cc:3037] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:07.867182 12001 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:07.872723 12142 raft_consensus.cc:1226] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 1. Current term is 2. Ops: []
W20260808 20:14:07.864305 14438 consensus_queue.cc:1159] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b9d94f35ea15421aa1d721201c782d8a" responder_term: 4 status { last_received { term: 4 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 4 index: 4 } }
I20260808 20:14:07.841398 14465 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:07.854498 14443 raft_consensus.cc:2726] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:07.850307 14380 consensus_queue.cc:1046] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260808 20:14:07.885960 14436 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:07.846426 14482 consensus_queue.cc:1046] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260808 20:14:07.855433 14424 consensus_queue.cc:1046] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260808 20:14:07.866631 14396 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:07.861804 14409 consensus_queue.cc:1046] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260808 20:14:07.874344 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:07.848268 14449 consensus_queue.cc:1035] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260808 20:14:07.893023 14217 consensus_queue.cc:1035] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260808 20:14:07.896050 14489 log.cc:924] Time spent T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
I20260808 20:14:07.896579 14465 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:07.879573 14457 raft_consensus.cc:686] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:07.886539 14436 raft_consensus.cc:2726] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:07.905408 12251 raft_consensus.cc:1261] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:14:07.905833 14344 tablet_copy_client.cc:531] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:14:07.920989 14460 consensus_queue.cc:1046] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:14:07.918711 12243 raft_consensus.cc:2379] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 2: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:07.910959 14428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:53290 (request call id 397) took 2031 ms. Trace:
I20260808 20:14:07.933800 14457 consensus_queue.cc:237] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:07.936793 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:07.944658 12251 raft_consensus.cc:2454] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:07.946943 11994 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:07.950251 14413 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:07.952087 14469 raft_consensus.cc:2781] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:07.952684 14469 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:07.953238 14469 raft_consensus.cc:3037] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:07.962251 12038 raft_consensus.cc:595] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:07.962670 12038 raft_consensus.cc:603] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 20:14:07.967964 11896 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:57500 (request call id 310) took 1868 ms. Trace:
I20260808 20:14:07.969060 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:07.970016 13072 raft_consensus.cc:2454] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:07.919968 14383 consensus_queue.cc:1035] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260808 20:14:07.971606 12207 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:07.972417 14452 raft_consensus.cc:2781] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:07.972965 14452 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:07.973493 14452 raft_consensus.cc:3037] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:07.981957 14430 raft_consensus.cc:3032] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:07.982538 14430 raft_consensus.cc:729] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:07.947962 14425 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:07.983796 14430 consensus_queue.cc:260] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:08.004817 12040 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "b9d94f35ea15421aa1d721201c782d8a"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:14:08.005514 12040 raft_consensus.cc:595] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Received request to transfer leadership to TS b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:08.011438 14450 raft_consensus.cc:3032] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:08.011984 14450 raft_consensus.cc:729] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:08.013120 14450 consensus_queue.cc:260] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:08.014356 14450 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:07.968309 11896 rpcz_store.cc:276] 0808 20:14:06.099370 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:06.099494 (+ 124us) service_pool.cc:227] Handling call
0808 20:14:07.967874 (+1868380us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 20:14:06.581547 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 20:14:06.582356 (+ 809us) write_op.cc:432] Acquiring schema lock in shared mode
0808 20:14:06.582412 (+ 56us) write_op.cc:435] Acquired schema lock
0808 20:14:06.582455 (+ 43us) tablet.cc:660] Decoding operations
0808 20:14:06.583023 (+ 568us) write_op.cc:620] Acquiring the partition lock for write op
0808 20:14:06.583160 (+ 137us) write_op.cc:641] Partition lock acquired for write op
0808 20:14:06.583210 (+ 50us) tablet.cc:683] Acquiring locks for 1 operations
0808 20:14:06.583617 (+ 407us) tablet.cc:699] Row locks acquired
0808 20:14:06.583681 (+ 64us) write_op.cc:260] PREPARE: finished
0808 20:14:06.583917 (+ 236us) write_op.cc:270] Start()
0808 20:14:06.584068 (+ 151us) write_op.cc:276] Timestamp: P: 1786220046583851 usec, L: 0
0808 20:14:06.584112 (+ 44us) op_driver.cc:331] REPLICATION: starting
0808 20:14:06.597400 (+ 13288us) log.cc:842] Serialized 535 byte log entry
0808 20:14:07.506996 (+909596us) spinlock_profiling.cc:243] Waited 7.89 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:07.731325 (+224329us) op_driver.cc:444] REPLICATION: finished
0808 20:14:07.890954 (+159629us) write_op.cc:301] APPLY: starting
0808 20:14:07.893074 (+ 2120us) tablet.cc:1365] starting BulkCheckPresence
0808 20:14:07.902439 (+ 9365us) tablet.cc:1368] finished BulkCheckPresence
0808 20:14:07.902461 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 20:14:07.902721 (+ 260us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 20:14:07.902859 (+ 138us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0808 20:14:07.902900 (+ 41us) write_op.cc:312] APPLY: finished
0808 20:14:07.903096 (+ 196us) log.cc:842] Serialized 23 byte log entry
0808 20:14:07.903246 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:14:07.903309 (+ 63us) write_op.cc:454] Released schema lock
0808 20:14:07.903346 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0808 20:14:07.903595 (+ 249us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:14:07.903626 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":143958,"apply.run_cpu_time_us":1705,"apply.run_wall_time_us":18654,"num_ops":1,"prepare.queue_time_us":299563,"prepare.run_cpu_time_us":90721,"prepare.run_wall_time_us":432690,"raft.queue_time_us":153371,"replication_time_us":1110275,"spinlock_wait_cycles":23879936,"thread_start_us":857784,"threads_started":4,"wal-append.queue_time_us":578879}]]}
I20260808 20:14:08.037673 14430 raft_consensus.cc:3037] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:07.938918 14428 rpcz_store.cc:276] 0808 20:14:05.877326 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:05.881300 (+ 3974us) service_pool.cc:227] Handling call
0808 20:14:06.043489 (+162189us) ts_tablet_manager.cc:928] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
0808 20:14:06.828259 (+784770us) tablet_copy_client.cc:361] Tablet copy session begun
0808 20:14:07.559211 (+730952us) tablet_metadata.cc:641] Metadata flushed
0808 20:14:07.562230 (+ 3019us) tablet_copy_client.cc:490] Wrote new tablet metadata
0808 20:14:07.842220 (+279990us) tablet_copy_client.cc:411] Wrote new consensus metadata
0808 20:14:07.857587 (+ 15367us) tablet_replica.cc:205] Creating consensus instance
0808 20:14:07.904615 (+ 47028us) ts_tablet_manager.cc:983] Replica registered.
0808 20:14:07.909009 (+ 4394us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":120439,"tablet-copy.queue_time_us":78454,"thread_start_us":171820,"threads_started":2}
I20260808 20:14:08.076036 14428 tablet_copy_client.cc:791] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 0 data blocks...
W20260808 20:14:08.080158 14492 log.cc:924] Time spent T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.202s user 0.001s sys 0.000s
W20260808 20:14:08.082147 14496 log.cc:924] Time spent T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
I20260808 20:14:08.045603 12137 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:49548
I20260808 20:14:08.076829 14428 tablet_copy_client.cc:655] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Starting download of 1 WAL segments...
I20260808 20:14:08.104735 14469 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:08.110798 14469 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:08.111977 12137 raft_consensus.cc:595] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:08.112607 12137 raft_consensus.cc:603] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 20:14:08.113215 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:08.121871 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
W20260808 20:14:08.122146 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 248486us
I20260808 20:14:08.123546 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:14:08.123509 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:08.124260 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:14:08.147405 12147 raft_consensus.cc:2454] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:08.149430 12000 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:08.098079 14396 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:08.157150 12241 raft_consensus.cc:1226] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:08.158854 14396 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:08.154196 14430 raft_consensus.cc:2781] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:14:08.159744 14430 raft_consensus.cc:686] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:08.160589 14430 consensus_queue.cc:237] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:08.139895 14414 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:08.172235 14351 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:08.182740 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:08.179987 14414 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:08.186690 14351 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:08.193414 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:08.194083 14344 tablet_bootstrap.cc:492] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:14:08.194128 12147 raft_consensus.cc:2379] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 2: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:08.198732 13076 raft_consensus.cc:2379] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 3: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:08.203660 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:08.194005 14351 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:08.283017 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 415888us
I20260808 20:14:08.287276 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:08.209951 14414 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:08.176052 14469 consensus_queue.cc:1046] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260808 20:14:08.293969 13078 raft_consensus.cc:2379] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 2: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:08.273363 12241 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:08.328935 12043 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:08.286536 14408 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:08.332062 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:08.332866 12038 raft_consensus.cc:2379] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 3: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:08.288527 14416 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:08.294872 14422 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:08.330047 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:08.335340 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:08.332767 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:08.335731 13074 raft_consensus.cc:2454] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:08.341662 13014 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:08.348222 13078 raft_consensus.cc:1261] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260808 20:14:08.357967 14511 log.cc:924] Time spent T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.075s user 0.000s sys 0.001s
I20260808 20:14:08.356070 14402 raft_consensus.cc:3032] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:08.360798 14402 raft_consensus.cc:729] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:08.360930 12047 raft_consensus.cc:1226] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:08.362229 14402 consensus_queue.cc:260] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:08.370774 14389 raft_consensus.cc:3032] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:08.371348 14389 raft_consensus.cc:729] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:08.374296 14445 raft_consensus.cc:482] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:08.372817 14389 consensus_queue.cc:260] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:08.374799 14445 raft_consensus.cc:504] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:08.375804 14389 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:08.349927 14420 raft_consensus.cc:2781] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:08.387964 14420 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:08.388540 14420 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:08.336653 14416 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:08.337409 14408 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:08.339218 14422 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:08.389786 14414 raft_consensus.cc:482] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:08.390167 14402 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:08.405205 12137 raft_consensus.cc:1261] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:14:08.426107 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:08.427055 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:08.429422 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:08.429813 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:08.439781 12046 raft_consensus.cc:2454] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:08.402895 14416 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:08.442270 12038 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:08.429466 12043 raft_consensus.cc:1226] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:08.435272 11889 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44508 (request call id 191) took 1241 ms. Trace:
I20260808 20:14:08.441124 12102 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:08.439874 12039 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:14:08.447079 13074 raft_consensus.cc:1261] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:14:08.451385 14457 consensus_queue.cc:1035] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:08.453920 14452 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:08.458544 14457 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:08.458947 12251 raft_consensus.cc:2379] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 3: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:08.467012 12205 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:08.473085 12205 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:08.439435 14445 leader_election.cc:288] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:08.474398 14428 tablet_copy_client.cc:531] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:14:08.471920 14416 raft_consensus.cc:2781] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:08.476222 14416 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:08.476754 14416 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:08.478088 14217 raft_consensus.cc:2781] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:08.478549 14217 raft_consensus.cc:686] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:08.459471 14457 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:08.487797 14217 consensus_queue.cc:237] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:08.443744 11889 rpcz_store.cc:276] 0808 20:14:07.193697 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:07.193843 (+ 146us) service_pool.cc:227] Handling call
0808 20:14:08.435201 (+1241358us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 20:14:07.345889 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 20:14:07.407962 (+ 62073us) write_op.cc:432] Acquiring schema lock in shared mode
0808 20:14:07.408008 (+ 46us) write_op.cc:435] Acquired schema lock
0808 20:14:07.408027 (+ 19us) tablet.cc:660] Decoding operations
0808 20:14:07.408458 (+ 431us) write_op.cc:620] Acquiring the partition lock for write op
0808 20:14:07.408527 (+ 69us) write_op.cc:641] Partition lock acquired for write op
0808 20:14:07.408577 (+ 50us) tablet.cc:683] Acquiring locks for 1 operations
0808 20:14:07.408927 (+ 350us) tablet.cc:699] Row locks acquired
0808 20:14:07.408969 (+ 42us) write_op.cc:260] PREPARE: finished
0808 20:14:07.409205 (+ 236us) write_op.cc:270] Start()
0808 20:14:07.409389 (+ 184us) write_op.cc:276] Timestamp: P: 1786220047409137 usec, L: 0
0808 20:14:07.409409 (+ 20us) op_driver.cc:331] REPLICATION: starting
0808 20:14:07.410326 (+ 917us) log.cc:842] Serialized 530 byte log entry
0808 20:14:08.167764 (+757438us) op_driver.cc:444] REPLICATION: finished
0808 20:14:08.168090 (+ 326us) write_op.cc:301] APPLY: starting
0808 20:14:08.168228 (+ 138us) tablet.cc:1365] starting BulkCheckPresence
0808 20:14:08.168364 (+ 136us) tablet.cc:1368] finished BulkCheckPresence
0808 20:14:08.181444 (+ 13080us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 20:14:08.193128 (+ 11684us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 20:14:08.193277 (+ 149us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0808 20:14:08.193331 (+ 54us) write_op.cc:312] APPLY: finished
0808 20:14:08.193535 (+ 204us) log.cc:842] Serialized 23 byte log entry
0808 20:14:08.193705 (+ 170us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:14:08.193781 (+ 76us) write_op.cc:454] Released schema lock
0808 20:14:08.193820 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0808 20:14:08.194119 (+ 299us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:14:08.194145 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":144,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":107,"apply.run_cpu_time_us":2135,"apply.run_wall_time_us":26250,"num_ops":1,"prepare.queue_time_us":2112,"prepare.run_cpu_time_us":3863,"prepare.run_wall_time_us":79563,"replication_time_us":758188}]]}
I20260808 20:14:08.495556 14408 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:08.466984 12036 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:08.539712 14450 consensus_queue.cc:1035] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:08.495594 14443 raft_consensus.cc:3032] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:08.541374 14443 raft_consensus.cc:729] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:08.477700 14426 tablet_copy_client.cc:531] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 20:14:08.437568 14414 raft_consensus.cc:504] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:08.479902 14341 log.cc:924] Time spent T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.091s user 0.001s sys 0.000s
I20260808 20:14:08.543689 12102 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:08.547551 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:08.486573 14433 raft_consensus.cc:2781] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:08.552861 12045 raft_consensus.cc:2379] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 3: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:08.553082 14433 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:08.555538 14433 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:08.556145 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:08.496397 14400 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:08.542992 14355 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:08.573365 14355 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:08.575377 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:08.576028 13076 raft_consensus.cc:2454] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:08.571550 14400 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:08.570737 14414 leader_election.cc:288] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:08.575518 14355 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:08.564563 12137 raft_consensus.cc:1226] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:08.577847 14400 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:08.578548 14400 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:08.576201 12243 raft_consensus.cc:2379] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 2: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:08.579285 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:08.589385 12045 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:08.581773 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:08.581938 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:08.545089 14452 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:08.564678 14351 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:08.590551 12102 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:08.593663 12137 raft_consensus.cc:2454] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:14:08.593587 14351 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:08.595264 13016 leader_election.cc:302] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:08.595615 14351 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:08.596450 14351 raft_consensus.cc:2781] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:08.596776 14351 raft_consensus.cc:482] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:08.597138 14351 raft_consensus.cc:3037] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:08.597209 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:08.602087 12137 raft_consensus.cc:1261] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:14:08.603916 14527 raft_consensus.cc:2781] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:08.604306 14527 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:08.604617 14527 raft_consensus.cc:3037] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:08.562701 14376 raft_consensus.cc:2781] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260808 20:14:08.607787 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 260547us
I20260808 20:14:08.562953 14375 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:08.616671 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:08.617288 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:08.618255 12249 raft_consensus.cc:2454] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:08.589736 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:08.620996 14522 consensus_queue.cc:1035] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260808 20:14:08.617306 14351 raft_consensus.cc:504] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:08.542399 14443 consensus_queue.cc:260] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:08.609519 14432 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:08.609875 14376 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
W20260808 20:14:08.638827 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 159599us
I20260808 20:14:08.593134 14400 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:08.647161 13015 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:08.627523 14533 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:08.651429 14533 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:08.644028 14376 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:08.646775 14501 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:08.645861 14457 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:08.661080 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:08.683421 12254 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:08.684494 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:08.685170 13075 raft_consensus.cc:3037] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:08.683647 14533 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:08.643857 14432 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:08.690440 14432 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:08.691140 12250 raft_consensus.cc:2379] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 5: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:08.637437 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:08.692399 12243 raft_consensus.cc:2454] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:08.694267 14400 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:08.694582 12144 raft_consensus.cc:1261] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:14:08.647694 14450 consensus_queue.cc:1046] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260808 20:14:08.661553 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:08.696399 12137 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:08.696907 14445 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:08.623031 14375 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:08.697419 14445 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:08.698813 14375 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:08.699393 14445 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:08.662034 14341 tablet_bootstrap.cc:492] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:14:08.701921 14341 tablet_bootstrap.cc:492] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:14:08.702801 14341 ts_tablet_manager.cc:1397] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 1.231s user 0.050s sys 0.023s
I20260808 20:14:08.650688 13072 raft_consensus.cc:2454] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:08.663929 12249 raft_consensus.cc:1261] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:14:08.665588 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:08.704746 13014 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:08.668552 11896 catalog_manager.cc:5495] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 2 to 3, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 3 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:08.706331 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:08.713433 12045 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:08.713131 11896 catalog_manager.cc:5495] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 2 to 4, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 4 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } }
I20260808 20:14:08.660290 14351 leader_election.cc:288] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:08.678046 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:08.715795 12043 raft_consensus.cc:2454] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:14:08.715332 11896 catalog_manager.cc:5495] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 1 to 2, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4). New cstate: current_term: 2 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:08.688534 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:08.696372 11994 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:08.711789 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:08.671002 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:08.706038 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:08.714711 12102 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:08.718709 12046 raft_consensus.cc:2454] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:08.718927 12040 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:08.717170 13015 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:08.718688 11994 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:08.720563 12102 leader_election.cc:302] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
W20260808 20:14:08.683732 14521 log.cc:924] Time spent T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.119s user 0.001s sys 0.002s
I20260808 20:14:08.721465 12205 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:08.722270 12241 raft_consensus.cc:2379] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 2: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:08.714509 14400 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:08.739298 12146 raft_consensus.cc:1261] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:14:08.717939 14409 raft_consensus.cc:2781] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:08.722339 14397 raft_consensus.cc:2781] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:08.704377 14535 raft_consensus.cc:2781] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:08.725860 14408 raft_consensus.cc:2781] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:08.725540 14460 raft_consensus.cc:2726] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:08.732206 14351 raft_consensus.cc:2781] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:08.761313 14351 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:08.761677 14351 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:08.725337 14469 raft_consensus.cc:2781] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:08.764487 14469 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:08.764838 14469 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:08.768590 14426 tablet_bootstrap.cc:492] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:14:08.757085 14535 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:08.769063 14535 raft_consensus.cc:3037] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:08.751592 14397 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:08.719623 14457 consensus_queue.cc:1035] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260808 20:14:08.731416 12048 raft_consensus.cc:2454] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:08.751058 14409 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:08.769557 14397 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:08.780682 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:08.774052 14376 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:08.776403 14396 consensus_queue.cc:1035] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:08.754333 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:08.704449 14501 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:08.787712 13078 raft_consensus.cc:2454] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:08.787863 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:08.759606 14408 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:08.754520 14538 raft_consensus.cc:2781] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:08.731096 14522 consensus_queue.cc:1035] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260808 20:14:08.745643 14416 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:08.788308 14409 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:08.810315 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:08.808530 12146 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:08.796612 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:08.796823 12207 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:08.810943 12250 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:08.805649 12036 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:08.813987 12138 raft_consensus.cc:2454] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:14:08.813864 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:08.817080 12241 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:08.821118 13016 leader_election.cc:302] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:08.790421 14480 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:08.748435 14400 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:08.823659 14428 tablet_bootstrap.cc:492] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap starting.
I20260808 20:14:08.802749 14408 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:08.828465 12147 raft_consensus.cc:2379] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:08.805071 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:08.805133 14538 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:08.837064 14351 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:08.839555 14217 raft_consensus.cc:2781] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:08.834262 14535 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:08.855877 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 66115.7us
I20260808 20:14:08.932549 14217 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:08.938395 14217 raft_consensus.cc:3037] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:08.838552 14468 raft_consensus.cc:2781] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:14:08.943382 12101 leader_election.cc:302] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:08.950264 13078 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:08.951903 14413 raft_consensus.cc:482] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:08.952531 14413 raft_consensus.cc:504] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:08.953662 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 131073us
I20260808 20:14:08.949957 14217 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:08.938659 14532 log.cc:924] Time spent T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.116s user 0.001s sys 0.008s
I20260808 20:14:08.931960 14538 raft_consensus.cc:3037] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:08.959015 14217 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:08.842662 14375 consensus_queue.cc:1035] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.111s
I20260808 20:14:08.838326 14480 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:08.962707 13071 raft_consensus.cc:2379] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 2: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:08.981498 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:08.991009 13073 raft_consensus.cc:3037] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:08.980613 14413 leader_election.cc:288] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:14:09.004568 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 292321us
I20260808 20:14:09.053427 14555 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:09.054037 14555 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:09.056262 14469 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:09.061038 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:09.005069 14376 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:08.995061 14388 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:09.074470 12040 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:14:09.078644 13015 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:09.085232 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:09.085801 12038 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
W20260808 20:14:09.089558 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 250439us
I20260808 20:14:08.990131 14341 raft_consensus.cc:348] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:09.071569 14422 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:09.075810 14555 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:09.094046 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:09.091866 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:09.073053 14416 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:09.093240 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:09.109025 12046 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:09.113584 12039 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:09.115402 13079 raft_consensus.cc:1261] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:14:09.115906 12102 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:09.083397 14540 raft_consensus.cc:482] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:09.080490 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:09.090893 14501 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:09.119194 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:09.122574 14556 log.cc:924] Time spent T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260808 20:14:09.123620 14480 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:09.097580 14535 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:09.097695 14449 raft_consensus.cc:2781] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:14:09.127252 12102 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
W20260808 20:14:09.144296 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 65749.2us
I20260808 20:14:09.106989 14351 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:09.073609 14388 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:09.104489 14341 raft_consensus.cc:729] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:14:09.120527 14413 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:09.105902 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:09.105772 13075 raft_consensus.cc:2454] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:09.119568 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:09.087046 14400 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:09.083348 14397 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:09.151070 12038 raft_consensus.cc:2454] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:09.135545 14563 raft_consensus.cc:2781] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:09.106161 14468 raft_consensus.cc:686] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
W20260808 20:14:09.162920 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 208132us
I20260808 20:14:09.151734 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:09.170734 12036 raft_consensus.cc:2454] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:14:09.151767 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:09.162211 14522 consensus_queue.cc:1035] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260808 20:14:09.158684 14469 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:09.193380 12207 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:09.167536 14563 raft_consensus.cc:482] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:09.191028 12243 raft_consensus.cc:2454] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:14:09.194242 14563 raft_consensus.cc:3037] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:09.195112 14569 raft_consensus.cc:2781] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260808 20:14:09.191644 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 248350us
I20260808 20:14:09.165267 14449 raft_consensus.cc:686] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:09.200759 14375 raft_consensus.cc:2781] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:09.199990 13015 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:09.136595 14540 raft_consensus.cc:504] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:09.194649 14400 raft_consensus.cc:2781] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:09.209764 13015 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:09.199044 14569 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:14:09.207989 14433 consensus_meta.cc:287] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.652s user 0.011s sys 0.000s
I20260808 20:14:09.164772 14413 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:09.197634 13073 raft_consensus.cc:2454] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:09.222335 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:09.204677 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:09.212148 14380 raft_consensus.cc:2781] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:09.222465 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:09.212317 14400 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:14:09.215903 14420 consensus_meta.cc:287] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.825s user 0.010s sys 0.003s
I20260808 20:14:09.227339 12250 raft_consensus.cc:2379] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 5: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:09.228207 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:09.232656 12251 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:09.214839 14383 raft_consensus.cc:2781] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:09.213423 14572 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:09.238770 12207 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:09.218503 14570 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:09.220506 14569 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:09.226723 14480 raft_consensus.cc:482] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:09.225713 14460 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:09.192948 14341 consensus_queue.cc:260] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:09.229604 14400 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:09.246483 14416 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:09.246963 14416 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:09.237723 14383 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:09.224745 14468 consensus_queue.cc:237] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:09.241923 14570 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:09.231988 14420 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:09.258504 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:09.260761 13016 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:09.232259 14580 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:09.242022 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:09.247857 14480 raft_consensus.cc:504] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:09.277842 12254 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:09.262280 14383 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:09.271292 14443 raft_consensus.cc:3037] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:09.272387 14576 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:09.273376 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:09.239514 14380 raft_consensus.cc:686] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:09.223220 14433 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:09.276443 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:09.277139 14571 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:09.277900 14413 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:09.240497 14572 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:09.273813 14428 tablet_bootstrap.cc:700] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent opening tablet: real 0.109s user 0.001s sys 0.000s
I20260808 20:14:09.248955 14435 raft_consensus.cc:2781] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:09.280682 14397 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:09.279744 14420 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:09.282356 14388 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:09.251207 14460 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:09.241657 14449 consensus_queue.cc:237] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:09.298205 14469 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:09.314002 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:09.314555 14435 raft_consensus.cc:686] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:09.316282 14435 consensus_queue.cc:237] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:09.317405 14469 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:09.300277 12036 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:09.333771 14570 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:09.334607 14570 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:09.335156 14570 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:09.355228 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:09.305742 14571 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:09.355896 12038 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:09.355754 14380 consensus_queue.cc:237] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:09.360333 12102 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:09.351665 12047 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:09.283146 14580 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:09.352434 14444 raft_consensus.cc:465] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Not starting pre-election -- already a leader
I20260808 20:14:09.301697 14576 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:09.357862 14341 ts_tablet_manager.cc:1428] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.655s user 0.009s sys 0.006s
W20260808 20:14:09.450507 14341 ts_tablet_manager.cc:1460] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20: Tablet startup took 747ms
W20260808 20:14:09.426121 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 622421us
W20260808 20:14:09.369284 14527 consensus_meta.cc:287] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.749s user 0.003s sys 0.011s
I20260808 20:14:09.451573 14522 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:09.451972 12036 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:09.452307 14522 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:09.457134 14433 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:09.458079 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:09.458941 14522 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:09.461380 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:09.461010 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:09.464022 12046 raft_consensus.cc:3037] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:09.467464 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:09.462329 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:09.473181 12045 raft_consensus.cc:2379] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 2: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:09.476234 12046 raft_consensus.cc:2454] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:09.483253 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:09.483801 12045 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:09.429342 14416 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:09.497090 12205 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:09.451261 14571 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:09.361562 14575 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:09.490948 12039 raft_consensus.cc:2454] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:09.411036 12254 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
W20260808 20:14:09.503545 14409 consensus_meta.cc:287] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.675s user 0.011s sys 0.000s
I20260808 20:14:09.508924 12040 raft_consensus.cc:2454] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:09.451858 14527 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:09.511730 14563 raft_consensus.cc:504] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:09.484203 14480 leader_election.cc:288] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:09.410557 14400 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:09.408049 14408 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:09.546324 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:09.450995 14341 ts_tablet_manager.cc:1462] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20: Trace:
0808 20:14:03.430289 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:03.433524 (+ 3235us) service_pool.cc:227] Handling call
0808 20:14:04.206631 (+773107us) ts_tablet_manager.cc:928] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
0808 20:14:05.844385 (+1637754us) tablet_copy_client.cc:361] Tablet copy session begun
0808 20:14:06.150003 (+305618us) tablet_metadata.cc:641] Metadata flushed
0808 20:14:06.166166 (+ 16163us) tablet_copy_client.cc:490] Wrote new tablet metadata
0808 20:14:06.384286 (+218120us) tablet_copy_client.cc:411] Wrote new consensus metadata
0808 20:14:06.384564 (+ 278us) tablet_replica.cc:205] Creating consensus instance
0808 20:14:06.384886 (+ 322us) ts_tablet_manager.cc:983] Replica registered.
0808 20:14:06.417873 (+ 32987us) inbound_call.cc:173] Queueing success response
0808 20:14:07.471861 (+1053988us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0808 20:14:08.703179 (+1231318us) ts_tablet_manager.cc:1429] Starting tablet replica
0808 20:14:08.840525 (+137346us) tablet_replica.cc:259] Starting instrumentation
0808 20:14:08.981414 (+140889us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":248,"mutex_wait_us":335,"raft.queue_time_us":11043,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":133,"tablet-copy.queue_time_us":772062,"tablet-download-pool.queue_time_us":176,"tablet-download-pool.run_cpu_time_us":2611,"tablet-download-pool.run_wall_time_us":102891,"thread_start_us":1210926,"threads_started":2}
I20260808 20:14:09.544808 14469 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:09.541085 14460 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:09.657294 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:09.658166 12138 raft_consensus.cc:2379] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:09.659482 14400 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:09.659298 14578 raft_consensus.cc:2781] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:09.697556 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:09.505370 14388 raft_consensus.cc:2781] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:09.500422 14570 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:09.513631 12141 raft_consensus.cc:2354] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3.
I20260808 20:14:09.638993 14522 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:09.732257 12205 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:09.730898 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:09.738113 12038 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:09.505926 14416 raft_consensus.cc:482] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:09.617906 14538 consensus_meta.cc:287] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.614s user 0.012s sys 0.000s
I20260808 20:14:09.730604 14408 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:09.735394 14580 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:09.512768 14527 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:09.710096 13015 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:09.576581 14501 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:09.648492 14351 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:09.612150 14560 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:09.546200 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:09.525740 14409 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:09.754156 12205 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:09.700783 14578 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:09.729362 14388 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:09.662950 14569 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:09.751451 14426 tablet_bootstrap.cc:700] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20: Time spent opening tablet: real 0.280s user 0.002s sys 0.000s
I20260808 20:14:09.773855 14501 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:09.801106 12039 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:09.766862 12137 raft_consensus.cc:2379] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:09.648862 14383 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:09.510249 14575 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:09.768286 13076 raft_consensus.cc:1226] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 3. Current term is 4. Ops: []
W20260808 20:14:09.828480 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 74405.9us
I20260808 20:14:09.755177 14396 raft_consensus.cc:2781] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260808 20:14:09.819921 14375 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.040s sys 0.099s
I20260808 20:14:09.769956 14563 leader_election.cc:288] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:09.838645 14460 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:09.774528 14572 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:09.746616 14416 raft_consensus.cc:504] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:09.842705 12205 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:09.827088 14578 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:09.746719 14538 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:09.829030 14388 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:09.849961 13080 raft_consensus.cc:2354] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for earlier term 2. Current term is 3.
I20260808 20:14:09.840586 14355 raft_consensus.cc:2781] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:09.850183 12205 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
W20260808 20:14:09.843819 14375 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.625s user 0.040s sys 0.100s
I20260808 20:14:09.852599 13078 raft_consensus.cc:1203] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:14:09.853425 12205 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:09.747845 14522 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:09.852193 14571 raft_consensus.cc:2781] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:09.834779 14397 raft_consensus.cc:2781] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:14:09.774701 14351 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:09.840757 14396 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:09.855155 14469 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:09.819939 14560 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:09.866478 14522 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:09.866860 14432 raft_consensus.cc:2781] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:09.868351 12207 leader_election.cc:398] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Vote denied by peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901) with higher term. Message: Invalid argument: T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for earlier term 2. Current term is 3.
I20260808 20:14:09.864080 14469 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:09.859743 14571 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:09.857566 14472 raft_consensus.cc:2781] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:09.840688 14460 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:09.858649 14375 raft_consensus.cc:686] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:09.862183 14396 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:09.852406 14444 consensus_queue.cc:1046] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260808 20:14:09.917655 14533 raft_consensus.cc:3032] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:09.905563 14571 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:09.861975 14355 raft_consensus.cc:686] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:09.912242 14351 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:09.881506 14432 raft_consensus.cc:686] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 5 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:09.929021 12043 raft_consensus.cc:1226] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:09.922242 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:09.880605 12041 consensus_meta.cc:287] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.690s user 0.012s sys 0.004s
I20260808 20:14:09.874958 14397 raft_consensus.cc:686] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:09.905136 14469 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:09.929975 14432 consensus_queue.cc:237] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:09.932924 14558 consensus_queue.cc:1046] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 20:14:09.933571 14383 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:09.906543 14472 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:09.877997 14575 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:09.882797 14576 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:09.922317 14533 raft_consensus.cc:729] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
W20260808 20:14:09.943069 14344 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.701s user 0.032s sys 0.052s
W20260808 20:14:09.944567 13077 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.697s user 0.021s sys 0.022s
I20260808 20:14:09.935369 14522 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:09.936069 14409 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:09.962220 12101 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.678s user 0.017s sys 0.048s
I20260808 20:14:09.955327 14472 raft_consensus.cc:3037] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:14:09.984944 12101 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.701s user 0.018s sys 0.048s
I20260808 20:14:09.917951 14375 consensus_queue.cc:237] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:09.964069 14344 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.723s user 0.033s sys 0.052s
W20260808 20:14:09.963824 13077 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.716s user 0.021s sys 0.022s
I20260808 20:14:09.964979 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:09.990424 12038 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:09.965389 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:09.993664 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:09.994366 12250 raft_consensus.cc:2379] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 5: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:09.994220 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:09.994907 12244 raft_consensus.cc:2379] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:09.988906 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:09.934280 12041 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:09.993958 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
W20260808 20:14:09.989377 12142 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.769s user 0.010s sys 0.036s
W20260808 20:14:10.004585 12142 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.777s user 0.010s sys 0.036s
W20260808 20:14:10.005591 12102 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.010s sys 0.053s
I20260808 20:14:09.985213 14576 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:09.997118 12144 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.699s user 0.008s sys 0.027s
W20260808 20:14:10.011142 12105 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.726s user 0.027s sys 0.037s
W20260808 20:14:10.013497 12105 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.729s user 0.027s sys 0.037s
W20260808 20:14:10.011857 13014 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.019s sys 0.016s
W20260808 20:14:10.014518 13014 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.706s user 0.019s sys 0.016s
I20260808 20:14:09.964185 14416 leader_election.cc:288] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:09.998951 14569 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:10.006162 12102 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.645s user 0.010s sys 0.053s
W20260808 20:14:10.017593 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 657312us
W20260808 20:14:09.996379 13072 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.014s sys 0.046s
I20260808 20:14:10.008729 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:09.996562 14422 raft_consensus.cc:2726] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:09.987525 14501 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:14:10.011085 12140 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.007s sys 0.011s
W20260808 20:14:10.015992 12206 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.784s user 0.031s sys 0.037s
I20260808 20:14:10.005160 12254 raft_consensus.cc:3037] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:09.988722 14533 consensus_queue.cc:260] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:09.984972 14522 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:10.005012 14538 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:10.008528 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:09.987321 14355 consensus_queue.cc:237] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:10.475157 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 342558us
I20260808 20:14:10.407667 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:10.475628 12251 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 3
W20260808 20:14:10.524458 12140 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.060s user 0.013s sys 0.016s
I20260808 20:14:10.418148 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:10.407467 13077 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53290 (request call id 428) took 1162 ms. Trace:
I20260808 20:14:10.528002 12248 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:10.528986 13016 leader_election.cc:398] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Vote denied by peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139) with higher term. Message: Invalid argument: T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3.
I20260808 20:14:10.020413 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:10.530613 12244 raft_consensus.cc:2454] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
W20260808 20:14:10.524205 12206 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.293s user 0.039s sys 0.040s
I20260808 20:14:10.531275 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
W20260808 20:14:10.531486 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 1.30353e+06us
W20260808 20:14:10.475986 12249 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.993s user 0.021s sys 0.014s
W20260808 20:14:10.532235 12249 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.049s user 0.022s sys 0.015s
W20260808 20:14:10.536322 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 60183.1us
I20260808 20:14:10.260767 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
W20260808 20:14:10.010671 14380 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.002s sys 0.019s
W20260808 20:14:10.012017 14468 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.731s user 0.004s sys 0.015s
I20260808 20:14:10.528087 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1481) took 1259 ms. Trace:
I20260808 20:14:10.541893 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:10.555269 13080 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
W20260808 20:14:10.416054 14435 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.078s user 0.008s sys 0.026s
I20260808 20:14:10.563313 12140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1478) took 1463 ms. Trace:
I20260808 20:14:10.078912 14560 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:10.258585 12266 tablet_copy_service.cc:342] P b9d94f35ea15421aa1d721201c782d8a: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-cb8b79bb1c59483e93af813653a73a31 received from {username='slave'} at 127.0.0.1:53030
I20260808 20:14:10.320834 14572 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:10.549615 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:10.449642 14606 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:10.549903 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:10.544660 14420 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:10.551469 14522 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:10.440995 14388 consensus_meta.cc:287] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.573s user 0.005s sys 0.006s
I20260808 20:14:10.012247 14576 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:10.611308 14576 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:10.611830 14576 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:10.154444 12247 raft_consensus.cc:2379] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:10.613276 14400 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:10.613773 14400 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:10.436013 14457 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.097s user 0.010s sys 0.026s
W20260808 20:14:10.444936 14443 consensus_meta.cc:287] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 1.155s user 0.009s sys 0.003s
I20260808 20:14:10.615120 14443 raft_consensus.cc:482] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:10.612321 14576 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:10.615556 14443 raft_consensus.cc:504] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:10.616194 14400 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:10.475754 13074 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.178s user 0.014s sys 0.022s
I20260808 20:14:10.617497 14443 leader_election.cc:288] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:10.524214 13072 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.161s user 0.024s sys 0.056s
W20260808 20:14:10.617805 13074 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.320s user 0.014s sys 0.022s
I20260808 20:14:10.618117 14388 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:10.617379 14576 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:10.012691 12144 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.719s user 0.008s sys 0.027s
I20260808 20:14:10.555030 12142 rpcz_store.cc:276] 0808 20:14:09.211502 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:09.211630 (+ 128us) service_pool.cc:227] Handling call
0808 20:14:09.211898 (+ 268us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:09.212104 (+ 206us) raft_consensus.cc:1520] Early marking committed up to index 2
0808 20:14:10.006242 (+794138us) log.cc:842] Serialized 14 byte log entry
0808 20:14:10.470718 (+464476us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:10.470775 (+ 57us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:14:10.470820 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:10.470892 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:10.471033 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2379,"prepare.queue_time_us":1140973,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":3970,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":32373,"thread_start_us":1250037,"threads_started":2}
I20260808 20:14:10.563689 12140 rpcz_store.cc:276] 0808 20:14:09.100277 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:09.100396 (+ 119us) service_pool.cc:227] Handling call
0808 20:14:09.100687 (+ 291us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:09.161011 (+ 60324us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:09.161035 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:09.161431 (+ 396us) log.cc:842] Serialized 26 byte log entry
0808 20:14:09.464707 (+303276us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b180036e560. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96e2518b1 00007fa96e25e661 00007fa965394345 00007fa96e1cba39 00007fa9653b18bb 00007fa9653afcbf 00007fa9653b1e22 00007fa967f5b5bb 00007fa967f61063 00007fa967f61411 00007fa9682ae5cf
0808 20:14:10.527658 (+1062951us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:10.527698 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:10.527760 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:10.527799 (+ 39us) raft_consensus.cc:1671] finished
0808 20:14:10.527810 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:10.563250 (+ 35440us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":66462,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":12710,"spinlock_wait_cycles":23163648,"thread_start_us":1122731,"threads_started":2,"wal-append.queue_time_us":1202422}
I20260808 20:14:10.619755 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:10.620146 14388 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:10.620231 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:10.528576 13077 rpcz_store.cc:276] 0808 20:14:09.224050 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:09.224182 (+ 132us) service_pool.cc:227] Handling call
0808 20:14:09.224551 (+ 369us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:09.224936 (+ 385us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:09.224965 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:09.230745 (+ 5780us) log.cc:842] Serialized 26 byte log entry
0808 20:14:09.990056 (+759311us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:09.990096 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:09.990165 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:10.386173 (+396008us) raft_consensus.cc:1671] finished
0808 20:14:10.386193 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:10.386335 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2374,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":9291,"thread_start_us":742372,"threads_started":1,"wal-append.queue_time_us":1057849}
I20260808 20:14:10.620350 12142 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:10.566298 12105 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:10.258967 14570 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:10.566953 12248 raft_consensus.cc:2454] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6.
I20260808 20:14:10.212028 14558 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:10.337774 14540 leader_election.cc:288] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:10.541121 14550 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.086s user 0.003s sys 0.025s
I20260808 20:14:10.568423 12101 leader_election.cc:302] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:10.351050 14563 raft_consensus.cc:482] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:10.286489 14376 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:10.541056 14501 raft_consensus.cc:482] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:10.154819 14469 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:10.436276 14413 raft_consensus.cc:482] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:10.521922 14569 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:10.298501 14583 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
W20260808 20:14:10.554260 14468 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.274s user 0.007s sys 0.021s
W20260808 20:14:10.547098 14380 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.087s user 0.009s sys 0.026s
I20260808 20:14:10.573493 13079 raft_consensus.cc:2354] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6.
I20260808 20:14:10.010542 14397 consensus_queue.cc:237] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:10.562431 14435 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.225s user 0.008s sys 0.030s
W20260808 20:14:10.611411 14578 consensus_meta.cc:287] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.764s user 0.004s sys 0.008s
I20260808 20:14:10.526902 14416 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:10.288355 14460 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:10.622087 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:10.625422 12251 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:10.625015 12102 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
W20260808 20:14:10.625818 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 56060.5us
I20260808 20:14:10.568060 14595 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:10.574036 12266 tablet_copy_service.cc:433] P b9d94f35ea15421aa1d721201c782d8a: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-cb8b79bb1c59483e93af813653a73a31 on tablet cb8b79bb1c59483e93af813653a73a31 with peer e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:10.678426 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 110099us
I20260808 20:14:10.616746 14400 raft_consensus.cc:482] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
W20260808 20:14:10.624625 14457 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.305s user 0.010s sys 0.030s
W20260808 20:14:10.660823 14550 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.234s user 0.003s sys 0.026s
I20260808 20:14:10.620740 14575 raft_consensus.cc:482] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:10.587599 14572 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:10.709373 14449 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.830s user 0.021s sys 0.046s
I20260808 20:14:10.683182 14538 raft_consensus.cc:2781] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:10.680882 14376 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:10.737191 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 205158us
I20260808 20:14:10.658736 14570 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:10.747072 12101 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:10.621649 14420 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:10.635303 14606 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:10.680706 14597 raft_consensus.cc:2781] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:10.723623 14468 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:10.661311 14558 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:10.796156 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:10.794828 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:10.687240 14569 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:10.681849 14501 raft_consensus.cc:504] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:10.683418 14469 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:10.687068 14413 raft_consensus.cc:504] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:10.932330 14607 log.cc:924] Time spent T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.186s user 0.000s sys 0.000s
I20260808 20:14:10.717558 14563 raft_consensus.cc:504] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:10.795246 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:10.745451 14400 raft_consensus.cc:504] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:10.790115 14435 raft_consensus.cc:482] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:10.716691 14583 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:11.072825 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 137197us
I20260808 20:14:11.073239 12140 raft_consensus.cc:2454] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:11.126456 14563 leader_election.cc:288] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:11.127341 14563 raft_consensus.cc:482] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:11.003796 14597 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:11.128252 14597 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:10.728318 14595 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:11.068811 13074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 309) took 1845 ms. Trace:
I20260808 20:14:11.075173 12105 leader_election.cc:398] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Vote denied by peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901) with higher term. Message: Invalid argument: T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6.
I20260808 20:14:10.723701 14416 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:10.852272 14449 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.973s user 0.021s sys 0.046s
W20260808 20:14:10.931044 14611 log.cc:924] Time spent T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.139s user 0.000s sys 0.000s
W20260808 20:14:10.932350 14396 consensus_meta.cc:287] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 1.019s user 0.010s sys 0.001s
I20260808 20:14:11.098999 13072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53290 (request call id 429) took 1812 ms. Trace:
I20260808 20:14:10.718796 14578 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:10.918290 14538 raft_consensus.cc:482] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:10.919262 14388 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:11.161365 14396 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
W20260808 20:14:11.068104 14344 log.cc:924] Time spent T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.499s user 0.000s sys 0.000s
I20260808 20:14:10.919866 14389 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:11.079263 14591 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
W20260808 20:14:10.991144 14609 log.cc:924] Time spent T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045: Append to log took a long time: real 0.073s user 0.000s sys 0.001s
I20260808 20:14:11.149230 12105 leader_election.cc:401] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260808 20:14:10.850437 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:11.169404 14400 leader_election.cc:288] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:11.163623 14608 log.cc:924] Time spent T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.514s user 0.000s sys 0.001s
I20260808 20:14:11.165478 14606 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:11.166708 14376 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:11.128115 14563 raft_consensus.cc:504] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:11.186544 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 561607us
I20260808 20:14:11.192588 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:11.078339 14468 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:11.229689 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 154635us
I20260808 20:14:11.137961 14617 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:11.138556 14569 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:11.235339 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:10.849941 14575 raft_consensus.cc:504] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:11.141341 14583 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:11.217693 12251 raft_consensus.cc:2379] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:11.203666 14388 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:11.220100 14396 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:11.224475 12147 raft_consensus.cc:2454] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
W20260808 20:14:11.227058 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 480030us
I20260808 20:14:11.180140 14538 raft_consensus.cc:3037] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:11.233707 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:11.134670 14435 raft_consensus.cc:504] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:11.162562 14603 log.cc:924] Time spent T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.420s user 0.001s sys 0.001s
I20260808 20:14:11.250983 12102 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:11.211979 14351 raft_consensus.cc:482] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
W20260808 20:14:11.253435 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 427593us
I20260808 20:14:11.219852 14501 leader_election.cc:288] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:11.261160 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:11.248739 12144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1480) took 2024 ms. Trace:
I20260808 20:14:11.239876 14416 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:11.252696 14628 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:11.267701 12248 raft_consensus.cc:3037] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:11.284096 13014 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:11.284348 14572 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:11.255244 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:11.286928 13014 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
W20260808 20:14:11.287343 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 164579us
I20260808 20:14:11.270743 12147 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:11.258852 13079 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:11.251461 14400 raft_consensus.cc:2781] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:11.211220 14389 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:11.255892 14606 raft_consensus.cc:2781] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:11.288214 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:11.289759 14625 raft_consensus.cc:482] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:11.284816 14414 raft_consensus.cc:2726] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:11.214265 14591 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:11.253285 14617 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
W20260808 20:14:11.397950 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 73701us
I20260808 20:14:11.429853 14620 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 6
W20260808 20:14:11.919296 14604 log.cc:924] Time spent T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.137s user 0.000s sys 0.001s
I20260808 20:14:11.919502 14625 raft_consensus.cc:504] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:11.920943 12249 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1588) took 2705 ms. Trace:
W20260808 20:14:11.401736 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 114868us
I20260808 20:14:11.290082 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:11.921739 14615 log.cc:924] Time spent T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.159s user 0.000s sys 0.000s
I20260808 20:14:11.290050 12248 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:11.944828 12152 raft_consensus.cc:2454] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:11.922542 12147 raft_consensus.cc:2454] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:11.172356 13072 rpcz_store.cc:276] 0808 20:14:09.286319 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:09.286476 (+ 157us) service_pool.cc:227] Handling call
0808 20:14:09.306422 (+ 19946us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:09.322924 (+ 16502us) raft_consensus.cc:1520] Early marking committed up to index 5
0808 20:14:10.618250 (+1295326us) log.cc:842] Serialized 14 byte log entry
0808 20:14:10.851366 (+233116us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:10.935423 (+ 84057us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:10.939873 (+ 4450us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:11.023814 (+ 83941us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:11.098904 (+ 75090us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":67617,"prepare.queue_time_us":1047290,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":69868,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":82293,"thread_start_us":1377160,"threads_started":2}
W20260808 20:14:11.541865 12105 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.24.4:36901 timed out after 2.015s (SENT) [suppressed 14 similar messages]
I20260808 20:14:11.921311 12249 rpcz_store.cc:276] 0808 20:14:09.215687 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:09.215812 (+ 125us) service_pool.cc:227] Handling call
0808 20:14:09.224854 (+ 9042us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:09.227735 (+ 2881us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:09.227762 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:09.447011 (+219249us) spinlock_profiling.cc:243] Waited 1.83 ms on lock 0x7b2c0011ebd8. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa966cbde11 00007fa966ccd442 00007fa968315c11 00007fa9682da58a 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772
0808 20:14:09.460933 (+ 13922us) log.cc:842] Serialized 26 byte log entry
0808 20:14:10.532575 (+1071642us) raft_consensus.cc:1631] Marking committed up to 1
0808 20:14:10.532612 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:10.532674 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:11.920589 (+1387915us) raft_consensus.cc:1671] finished
0808 20:14:11.920608 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:11.920834 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2525,"raft.queue_time_us":6905,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":2908,"spinlock_wait_cycles":4205312,"thread_start_us":1049232,"threads_started":1,"wal-append.queue_time_us":1263262}
W20260808 20:14:11.861287 14593 log.cc:924] Time spent T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.467s user 0.000s sys 0.001s
I20260808 20:14:11.930460 14375 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:11.509781 14614 raft_consensus.cc:2781] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election won for term 6
W20260808 20:14:11.951337 12147 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1483) took 1429 ms (client timeout 1726 ms). Trace:
W20260808 20:14:11.429770 14619 log.cc:924] Time spent T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.218s user 0.000s sys 0.001s
W20260808 20:14:11.929740 14571 consensus_meta.cc:287] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 2.011s user 0.013s sys 0.001s
W20260808 20:14:11.967010 12152 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1189) took 1916 ms (client timeout 1626 ms). Trace:
W20260808 20:14:11.972260 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 430404us
W20260808 20:14:11.976260 12152 rpcz_store.cc:269] 0808 20:14:10.035058 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:10.067672 (+ 32614us) service_pool.cc:227] Handling call
0808 20:14:11.289943 (+1222271us) spinlock_profiling.cc:243] Waited 60.2 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:11.951026 (+661083us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":138354048}
I20260808 20:14:11.861269 14480 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:11.931936 14400 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:11.434300 14469 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:11.358809 14628 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:11.978564 14610 log.cc:924] Time spent T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.740s user 0.000s sys 0.001s
I20260808 20:14:11.979555 14631 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:11.980075 14631 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:11.923424 14595 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:11.945252 14578 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:11.866791 14570 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:11.859234 14583 raft_consensus.cc:482] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:11.888070 14630 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
W20260808 20:14:11.969313 12147 rpcz_store.cc:269] 0808 20:14:10.522329 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:11.287987 (+765658us) service_pool.cc:227] Handling call
0808 20:14:11.951281 (+663294us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:11.919123 14413 leader_election.cc:288] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:11.929518 14558 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:11.374578 14416 raft_consensus.cc:482] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:11.918891 14634 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:11.416445 14380 raft_consensus.cc:482] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:11.299383 14572 raft_consensus.cc:482] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:11.359136 14420 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:11.922063 14625 leader_election.cc:288] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:11.271755 14351 raft_consensus.cc:504] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:11.962544 14606 raft_consensus.cc:686] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:11.973531 14563 leader_election.cc:288] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:12.135690 14595 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:12.155851 14595 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:12.159600 14595 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:12.159742 14351 leader_election.cc:288] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:12.132632 14435 leader_election.cc:288] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:12.161423 14606 consensus_queue.cc:237] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:11.149875 13074 rpcz_store.cc:276] 0808 20:14:09.222982 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:09.225463 (+ 2481us) service_pool.cc:227] Handling call
0808 20:14:09.243340 (+ 17877us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:09.262851 (+ 19511us) raft_consensus.cc:1520] Early marking committed up to index 5
0808 20:14:10.618388 (+1355537us) log.cc:842] Serialized 14 byte log entry
0808 20:14:10.766507 (+148119us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:10.791569 (+ 25062us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:10.851750 (+ 60181us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:10.934975 (+ 83225us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:11.026019 (+ 91044us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39602,"prepare.queue_time_us":1104628,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":72,"raft.queue_time_us":14786,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":3531,"thread_start_us":1421482,"threads_started":2,"wal-append.queue_time_us":591275}
I20260808 20:14:11.972944 14375 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:12.166601 12039 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:12.168066 14375 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:11.980850 14628 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:11.982542 14631 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:14:12.170877 14389 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:14:12.125509 12103 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.023s user 0.000s sys 0.016s
W20260808 20:14:11.981792 14627 log.cc:924] Time spent T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.723s user 0.001s sys 0.001s
I20260808 20:14:12.104924 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:12.215894 12152 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 205) took 1642 ms (client timeout 1752 ms). Trace:
W20260808 20:14:12.217078 12152 rpcz_store.cc:269] 0808 20:14:10.573797 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:12.017355 (+1443558us) service_pool.cc:227] Handling call
0808 20:14:12.215832 (+198477us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:11.975474 14446 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
W20260808 20:14:12.211417 14624 log.cc:924] Time spent T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.912s user 0.000s sys 0.001s
W20260808 20:14:12.241711 12103 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.250s user 0.000s sys 0.018s
I20260808 20:14:12.250773 14628 raft_consensus.cc:482] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:12.017879 14468 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:12.251597 14628 raft_consensus.cc:504] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:11.278838 12144 rpcz_store.cc:276] 0808 20:14:09.210864 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:09.212012 (+ 1148us) service_pool.cc:227] Handling call
0808 20:14:09.239734 (+ 27722us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:09.249086 (+ 9352us) raft_consensus.cc:1520] Early marking committed up to index 5
0808 20:14:10.660638 (+1411552us) log.cc:842] Serialized 14 byte log entry
0808 20:14:11.180111 (+519473us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:11.204428 (+ 24317us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:11.211243 (+ 6815us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:11.222191 (+ 10948us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:11.235532 (+ 13341us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15217,"prepare.queue_time_us":1078654,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":5059,"raft.queue_time_us":142043,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":11763,"thread_start_us":1791584,"threads_started":3,"wal-append.queue_time_us":536673}
I20260808 20:14:11.982769 14629 raft_consensus.cc:2781] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:12.123025 14617 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:12.262913 14629 raft_consensus.cc:686] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:12.126703 14388 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:12.265283 14563 raft_consensus.cc:482] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:12.266052 14563 raft_consensus.cc:504] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:12.226028 14380 raft_consensus.cc:504] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:12.225811 14572 raft_consensus.cc:504] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:12.225634 14416 raft_consensus.cc:504] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:12.123220 12254 consensus_meta.cc:287] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 1.575s user 0.009s sys 0.001s
W20260808 20:14:12.139592 13075 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 564) took 2948 ms (client timeout 2015 ms). Trace:
W20260808 20:14:12.147100 14472 consensus_meta.cc:287] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 2.142s user 0.010s sys 0.003s
W20260808 20:14:12.136463 14623 log.cc:924] Time spent T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.695s user 0.001s sys 0.000s
I20260808 20:14:12.132354 14591 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:12.253976 14639 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:12.262130 14569 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:12.314884 14472 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:12.046483 14414 raft_consensus.cc:482] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:12.322057 14569 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:12.322503 14414 raft_consensus.cc:504] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:12.325449 14569 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:12.024679 14400 raft_consensus.cc:3037] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:12.018179 14376 raft_consensus.cc:482] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:12.329263 14376 raft_consensus.cc:504] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:12.037403 14614 raft_consensus.cc:686] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:12.265820 14446 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:12.255095 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1187) took 2203 ms. Trace:
I20260808 20:14:17.186914 12146 rpcz_store.cc:276] 0808 20:14:10.021424 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:10.024644 (+ 3220us) service_pool.cc:227] Handling call
0808 20:14:10.024972 (+ 328us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:10.025277 (+ 305us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:10.025302 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:10.421663 (+396361us) log.cc:842] Serialized 26 byte log entry
0808 20:14:10.724837 (+303174us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:10.724877 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:10.724963 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:12.225052 (+1500089us) raft_consensus.cc:1671] finished
0808 20:14:12.225076 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:12.225217 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":54602,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":67887,"thread_start_us":179687,"threads_started":1,"wal-append.queue_time_us":735076,"wal-append.run_cpu_time_us":4231,"wal-append.run_wall_time_us":2071581}
I20260808 20:14:10.205364 12038 raft_consensus.cc:2454] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
W20260808 20:14:17.189180 12146 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1486) took 6666 ms (client timeout 2835 ms). Trace:
W20260808 20:14:14.518581 12078 debug-util.cc:398] Leaking SignalData structure 0x7b08001e0940 after lost signal to thread 10340
I20260808 20:14:12.286670 12254 raft_consensus.cc:2454] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:17.213398 14344 tablet_bootstrap.cc:492] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replaying log segment 1/1 (209B/548B this segment, stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20260808 20:14:17.213892 12254 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1678) took 7225 ms (client timeout 1625 ms). Trace:
W20260808 20:14:17.214324 12254 rpcz_store.cc:269] 0808 20:14:09.988325 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:09.988475 (+ 150us) service_pool.cc:227] Handling call
0808 20:14:17.213698 (+7225223us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:12.305583 13075 rpcz_store.cc:269] 0808 20:14:09.190942 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:09.195319 (+ 4377us) service_pool.cc:227] Handling call
0808 20:14:12.139312 (+2943993us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b5400652054. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:12.139500 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3434987904}
I20260808 20:14:12.348003 14376 leader_election.cc:288] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:12.350135 14380 leader_election.cc:288] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:17.226323 11896 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:57488 (request call id 318) took 9105 ms. Trace:
I20260808 20:14:12.333704 14414 leader_election.cc:288] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:12.370266 14416 leader_election.cc:288] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:12.369985 14446 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:12.281383 14468 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:12.334614 14472 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:12.024055 14480 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:17.235818 14480 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:14:12.284245 14538 consensus_meta.cc:287] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 1.018s user 0.011s sys 0.000s
I20260808 20:14:17.238348 14538 raft_consensus.cc:504] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:12.285300 14563 leader_election.cc:288] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:11.984678 14571 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:12.317654 14572 leader_election.cc:288] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:12.226251 14583 raft_consensus.cc:504] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:12.284616 14597 consensus_meta.cc:287] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 1.156s user 0.013s sys 0.000s
I20260808 20:14:17.247561 14583 leader_election.cc:288] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:12.361295 14614 consensus_queue.cc:237] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:17.247258 14597 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:17.249907 14597 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:12.341495 14628 leader_election.cc:288] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:12.264417 14629 consensus_queue.cc:237] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:12.226398 14630 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:17.257407 14630 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:12.249037 14631 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:17.257679 12142 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:17.258390 14631 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:17.260167 12142 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 203) took 6713 ms (client timeout 1652 ms). Trace:
I20260808 20:14:12.210803 14634 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:17.260583 12142 rpcz_store.cc:269] 0808 20:14:10.556348 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:10.621856 (+ 65508us) service_pool.cc:227] Handling call
0808 20:14:17.259940 (+6638084us) spinlock_profiling.cc:243] Waited 6.54 s on lock 0x7b5400326c54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:17.260096 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15032630144}
W20260808 20:14:17.262203 12142 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 210) took 5365 ms (client timeout 1518 ms). Trace:
I20260808 20:14:17.262880 14634 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:12.322057 14639 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:13.640305 14641 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 1.309s user 0.001s sys 0.012s
W20260808 20:14:17.266407 14641 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 4.957s user 0.002s sys 0.012s
W20260808 20:14:13.613016 14647 log.cc:924] Time spent T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.267s user 0.001s sys 0.000s
I20260808 20:14:12.371559 14620 raft_consensus.cc:2726] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901) with higher term. Message: Invalid argument: T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6.
I20260808 20:14:17.270946 14639 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:17.272746 14571 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:17.260859 14631 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:15.056224 14668 log.cc:924] Time spent T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045: Append to log took a long time: real 1.277s user 0.001s sys 0.000s
I20260808 20:14:17.242057 14538 leader_election.cc:288] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:15.147378 14669 raft_consensus.cc:1050] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:14:16.318617 14670 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:16.374861 11994 connection.cc:232] RPC call timeout handler was delayed by -5.5615424101706594s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1554s for a response
I20260808 20:14:16.478331 12036 raft_consensus.cc:2454] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:17.275928 14670 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:17.277083 12036 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1156) took 7775 ms (client timeout 2726 ms). Trace:
W20260808 20:14:17.277505 12036 rpcz_store.cc:269] 0808 20:14:09.501257 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:09.504861 (+ 3604us) service_pool.cc:227] Handling call
0808 20:14:17.276851 (+7771990us) spinlock_profiling.cc:243] Waited 6.66 s on lock 0x7b540013e554. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:17.276996 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15312007936}
I20260808 20:14:16.520241 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.278710 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:14:16.641300 12241 raft_consensus.cc:2379] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
W20260808 20:14:16.656476 12206 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 4.637s user 0.062s sys 0.116s
W20260808 20:14:16.428472 12039 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49226 (request call id 297) took 6463 ms (client timeout 1652 ms). Trace:
W20260808 20:14:16.690037 13015 connection.cc:232] RPC call timeout handler was delayed by -3.1783295061904937s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15280000000000002s for a response
W20260808 20:14:17.280287 12206 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 5.261s user 0.063s sys 0.116s
W20260808 20:14:12.361140 14397 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.192s user 0.033s sys 0.068s
W20260808 20:14:17.280927 14397 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 6.111s user 0.047s sys 0.086s
I20260808 20:14:16.793197 14673 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:17.282205 14673 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:16.799552 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:17.280512 13015 connection.cc:232] RPC call timeout handler was delayed by -3.132907385006547s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1771s for a response
W20260808 20:14:17.283571 13015 connection.cc:232] RPC call timeout handler was delayed by -2.838617566972971s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18610000000000002s for a response
I20260808 20:14:17.283741 12045 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:17.278966 12041 raft_consensus.cc:2454] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 1.
I20260808 20:14:17.283917 14444 raft_consensus.cc:482] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:17.283788 14531 raft_consensus.cc:482] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:16.828127 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:16.859650 13016 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 4.970s user 0.041s sys 0.053s
I20260808 20:14:17.284009 14446 raft_consensus.cc:482] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:17.284920 14444 raft_consensus.cc:504] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:16.935817 12150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1190) took 6867 ms. Trace:
W20260808 20:14:16.946617 13014 connection.cc:232] RPC call timeout handler was delayed by -3.4241106559056789s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15180000000000002s for a response
W20260808 20:14:17.283950 12039 rpcz_store.cc:269] 0808 20:14:09.965065 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:09.965178 (+ 113us) service_pool.cc:227] Handling call
0808 20:14:16.427886 (+6462708us) spinlock_profiling.cc:243] Waited 2.17 s on lock 0x7b540034ec54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:16.428189 (+ 303us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4998699776}
W20260808 20:14:17.286242 13014 connection.cc:232] RPC call timeout handler was delayed by -3.2135106560308486s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17520000000000002s for a response
I20260808 20:14:16.991366 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
W20260808 20:14:17.286540 13014 connection.cc:232] RPC call timeout handler was delayed by -2.9660106559749693s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20270000000000002s for a response
W20260808 20:14:17.286842 13014 connection.cc:232] RPC call timeout handler was delayed by -2.8506076990161091s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1708s for a response
W20260808 20:14:17.287204 13014 connection.cc:232] RPC call timeout handler was delayed by -2.5031450458336622s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2156s for a response
I20260808 20:14:17.043705 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.047577 14672 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:17.289247 12046 raft_consensus.cc:2454] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 1.
I20260808 20:14:17.284934 14531 raft_consensus.cc:504] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:17.282989 14670 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:17.285778 13016 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 5.396s user 0.041s sys 0.053s
I20260808 20:14:17.289390 14672 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:17.290337 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 5.42896e+06us
W20260808 20:14:17.285004 12206 connection.cc:232] RPC call timeout handler was delayed by -1.3113327629398555s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1626s for a response
I20260808 20:14:17.290261 14670 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:17.292516 12206 connection.cc:232] RPC call timeout handler was delayed by -1.0845327628776431s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1878s for a response
I20260808 20:14:17.055243 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:17.064637 14606 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 4.756s user 0.011s sys 0.037s
I20260808 20:14:17.071058 14680 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
W20260808 20:14:17.074898 12147 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 204) took 6518 ms (client timeout 2142 ms). Trace:
W20260808 20:14:17.293555 12147 rpcz_store.cc:269] 0808 20:14:10.556666 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.074415 (+6517749us) service_pool.cc:214] Skipping call since client already timed out
0808 20:14:17.074668 (+ 253us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260808 20:14:17.294543 12147 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1193) took 5315 ms (client timeout 1610 ms). Trace:
W20260808 20:14:17.294880 12147 rpcz_store.cc:269] 0808 20:14:11.979156 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.294329 (+5315173us) service_pool.cc:214] Skipping call since client already timed out
0808 20:14:17.294489 (+ 160us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260808 20:14:17.295706 12147 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1488) took 6772 ms (client timeout 3216 ms). Trace:
W20260808 20:14:17.087803 14636 log.cc:924] Time spent T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 5.192s user 0.016s sys 0.027s
W20260808 20:14:17.091987 12203 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 5.045s user 0.023s sys 0.052s
I20260808 20:14:17.296042 14397 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:17.100653 14682 raft_consensus.cc:482] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:17.296722 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.099936 14575 leader_election.cc:288] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:17.105350 14428 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 4.930s user 0.181s sys 0.322s
I20260808 20:14:17.285800 14446 raft_consensus.cc:504] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:17.291827 14531 leader_election.cc:288] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:17.292914 14670 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:17.298336 14680 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:17.299059 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.299466 14446 leader_election.cc:288] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:17.300153 14670 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:17.110917 12207 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 5.977s user 0.029s sys 0.066s
I20260808 20:14:17.300882 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:17.301515 12207 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 6.167s user 0.029s sys 0.067s
I20260808 20:14:17.117126 14533 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:17.117628 14684 raft_consensus.cc:482] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:17.295195 14444 leader_election.cc:288] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:17.301760 12244 raft_consensus.cc:2379] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:17.302465 14684 raft_consensus.cc:504] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:17.303452 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.304122 12244 raft_consensus.cc:2454] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
W20260808 20:14:17.120015 13079 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53290 (request call id 430) took 6386 ms (client timeout 2835 ms). Trace:
I20260808 20:14:17.300402 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:17.122696 12140 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1489) took 6577 ms (client timeout 1610 ms). Trace:
W20260808 20:14:17.284988 13015 connection.cc:232] RPC call timeout handler was delayed by -2.8196273851208389s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1708s for a response
W20260808 20:14:17.305451 12140 rpcz_store.cc:269] 0808 20:14:10.544935 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:10.619993 (+ 75058us) service_pool.cc:227] Handling call
0808 20:14:17.122362 (+6502369us) spinlock_profiling.cc:243] Waited 452 ms on lock 0x7b540030c854. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:17.122521 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1040256384}
W20260808 20:14:17.306702 12140 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1197) took 5326 ms (client timeout 1818 ms). Trace:
W20260808 20:14:17.307039 12140 rpcz_store.cc:269] 0808 20:14:11.980248 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.306494 (+5326246us) service_pool.cc:214] Skipping call since client already timed out
0808 20:14:17.306638 (+ 144us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260808 20:14:17.307854 12140 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 209) took 5411 ms (client timeout 2027 ms). Trace:
W20260808 20:14:17.308202 12140 rpcz_store.cc:269] 0808 20:14:11.896130 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.307674 (+5411544us) service_pool.cc:214] Skipping call since client already timed out
0808 20:14:17.307793 (+ 119us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260808 20:14:17.126056 12248 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1686) took 6331 ms (client timeout 3219 ms). Trace:
W20260808 20:14:17.131162 12001 connection.cc:232] RPC call timeout handler was delayed by -4.6245428449474275s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.28350000000000003s for a response
W20260808 20:14:17.309064 12248 rpcz_store.cc:269] 0808 20:14:10.794854 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:10.794983 (+ 129us) service_pool.cc:227] Handling call
0808 20:14:17.125625 (+6330642us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b540029ed54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:17.125805 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":354850048}
I20260808 20:14:17.310361 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.311614 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.312325 12248 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:14:17.133715 12250 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1688) took 6338 ms. Trace:
I20260808 20:14:17.134591 14686 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:17.144870 12102 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.24.1:40747 timed out after 2.363s (SENT) [suppressed 9 similar messages]
I20260808 20:14:17.152043 13080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 318) took 6595 ms. Trace:
I20260808 20:14:17.154053 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:17.314631 12138 raft_consensus.cc:2396] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260808 20:14:17.315110 12138 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1185) took 7298 ms (client timeout 1878 ms). Trace:
W20260808 20:14:17.158715 11995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 7.010s user 0.042s sys 0.068s
W20260808 20:14:17.315891 14606 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 5.007s user 0.011s sys 0.037s
W20260808 20:14:17.165061 12000 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 6.921s user 0.033s sys 0.088s
I20260808 20:14:17.316383 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:17.316756 12000 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 7.072s user 0.033s sys 0.089s
I20260808 20:14:17.166631 12141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1183) took 7152 ms. Trace:
I20260808 20:14:17.171388 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.180809 12137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1184) took 7164 ms. Trace:
I20260808 20:14:17.316973 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:17.181110 12251 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1685) took 6386 ms (client timeout 3174 ms). Trace:
I20260808 20:14:17.305586 12247 raft_consensus.cc:2379] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
W20260808 20:14:17.317659 12251 rpcz_store.cc:269] 0808 20:14:10.794439 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:10.794584 (+ 145us) service_pool.cc:227] Handling call
0808 20:14:17.180659 (+6386075us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b5400086e54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:17.180918 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":327266048}
W20260808 20:14:17.189466 12038 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49226 (request call id 296) took 7224 ms (client timeout 1893 ms). Trace:
W20260808 20:14:17.296635 12203 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 5.250s user 0.023s sys 0.052s
W20260808 20:14:17.318571 12038 rpcz_store.cc:269] 0808 20:14:09.964576 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:09.964703 (+ 127us) service_pool.cc:227] Handling call
0808 20:14:17.189290 (+7224587us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:17.189826 14681 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:17.190864 12078 debug-util.cc:398] Leaking SignalData structure 0x7b080010fc80 after lost signal to thread 11946
I20260808 20:14:17.319926 14673 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:17.319808 14681 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:17.189608 12146 rpcz_store.cc:269] 0808 20:14:10.523072 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.188938 (+6665866us) service_pool.cc:214] Skipping call since client already timed out
0808 20:14:17.189100 (+ 162us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260808 20:14:17.320780 12078 debug-util.cc:398] Leaking SignalData structure 0x7b0800159560 after lost signal to thread 12083
I20260808 20:14:17.296743 14397 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:17.321133 12078 debug-util.cc:398] Leaking SignalData structure 0x7b08003261a0 after lost signal to thread 12084
W20260808 20:14:17.321393 12146 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1194) took 5342 ms (client timeout 2929 ms). Trace:
W20260808 20:14:17.321712 12146 rpcz_store.cc:269] 0808 20:14:11.979473 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.321220 (+5341747us) service_pool.cc:214] Skipping call since client already timed out
0808 20:14:17.321336 (+ 116us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260808 20:14:17.322337 12078 debug-util.cc:398] Leaking SignalData structure 0x7b080038d4a0 after lost signal to thread 12185
W20260808 20:14:17.205605 14400 consensus_meta.cc:287] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 4.879s user 0.021s sys 0.038s
W20260808 20:14:17.322512 12146 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1196) took 5342 ms (client timeout 3093 ms). Trace:
I20260808 20:14:17.322816 14400 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:17.323688 12243 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1677) took 7336 ms (client timeout 2929 ms). Trace:
W20260808 20:14:17.324069 12243 rpcz_store.cc:269] 0808 20:14:09.987730 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:09.990060 (+ 2330us) service_pool.cc:227] Handling call
0808 20:14:17.170988 (+7180928us) spinlock_profiling.cc:243] Waited 2.92 ms on lock 0x7b58001a1398. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:17.323620 (+152632us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6708992}
W20260808 20:14:17.212188 12101 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 5.240s user 0.017s sys 0.066s
W20260808 20:14:17.212747 12205 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 4.972s user 0.020s sys 0.039s
I20260808 20:14:17.216641 14344 tablet_bootstrap.cc:492] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:14:17.292697 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:17.328459 12205 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 5.088s user 0.021s sys 0.039s
W20260808 20:14:17.328969 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 5.14889e+06us
I20260808 20:14:17.329159 14344 tablet_bootstrap.cc:492] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:14:17.329600 14397 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:17.328428 13074 raft_consensus.cc:2454] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:14:17.330210 14344 ts_tablet_manager.cc:1397] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 9.136s user 0.081s sys 0.095s
I20260808 20:14:17.300503 14680 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:17.291920 14672 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:17.331436 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.332044 12248 raft_consensus.cc:2454] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:17.331786 14669 consensus_queue.cc:237] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:17.330580 12140 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1199) took 5345 ms (client timeout 2726 ms). Trace:
I20260808 20:14:17.222657 11889 catalog_manager.cc:5495] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 1 to 2, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260808 20:14:17.332921 12140 rpcz_store.cc:269] 0808 20:14:11.980828 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.308813 (+5327985us) service_pool.cc:214] Skipping call since client already timed out
0808 20:14:17.326582 (+ 17769us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 20:14:17.333189 14686 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:17.333808 12140 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1205) took 5180 ms (client timeout 3321 ms). Trace:
I20260808 20:14:17.226794 11896 rpcz_store.cc:276] 0808 20:14:08.120531 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:08.120666 (+ 135us) service_pool.cc:227] Handling call
0808 20:14:17.226061 (+9105395us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 20:14:08.957210 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 20:14:08.957722 (+ 512us) write_op.cc:432] Acquiring schema lock in shared mode
0808 20:14:08.957760 (+ 38us) write_op.cc:435] Acquired schema lock
0808 20:14:08.957775 (+ 15us) tablet.cc:660] Decoding operations
0808 20:14:09.003895 (+ 46120us) write_op.cc:620] Acquiring the partition lock for write op
0808 20:14:09.049975 (+ 46080us) write_op.cc:641] Partition lock acquired for write op
0808 20:14:09.058489 (+ 8514us) tablet.cc:683] Acquiring locks for 3 operations
0808 20:14:09.104462 (+ 45973us) tablet.cc:699] Row locks acquired
0808 20:14:09.104479 (+ 17us) write_op.cc:260] PREPARE: finished
0808 20:14:09.411562 (+307083us) spinlock_profiling.cc:243] Waited 4.63 ms on lock 0x7b2c00090c58. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa966cbde11 00007fa966cccada 00007fa966ccc763 00007fa966ccc701 00007fa968315b12 00007fa9683158ed 00007fa96bce97a0 00007fa96bce92b8 00007fa96bcedba1 00007fa96bcedb59 00007fa96bcedae9
0808 20:14:09.411643 (+ 81us) write_op.cc:270] Start()
0808 20:14:09.411977 (+ 334us) write_op.cc:276] Timestamp: P: 1786220049119310 usec, L: 0
0808 20:14:09.412006 (+ 29us) op_driver.cc:331] REPLICATION: starting
0808 20:14:09.426316 (+ 14310us) log.cc:842] Serialized 1289 byte log entry
0808 20:14:12.131525 (+2705209us) op_driver.cc:444] REPLICATION: finished
0808 20:14:13.677182 (+1545657us) write_op.cc:301] APPLY: starting
0808 20:14:13.677277 (+ 95us) tablet.cc:1365] starting BulkCheckPresence
0808 20:14:13.677420 (+ 143us) tablet.cc:1368] finished BulkCheckPresence
0808 20:14:13.677451 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 20:14:13.748143 (+ 70692us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 20:14:13.748347 (+ 204us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0808 20:14:13.748401 (+ 54us) write_op.cc:312] APPLY: finished
0808 20:14:13.777341 (+ 28940us) log.cc:842] Serialized 35 byte log entry
0808 20:14:13.778925 (+ 1584us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:14:15.049494 (+1270569us) write_op.cc:454] Released schema lock
0808 20:14:15.049627 (+ 133us) write_op.cc:341] FINISH: Updating metrics
0808 20:14:15.055398 (+ 5771us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:14:15.055436 (+ 38us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":44,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":1417039,"apply.run_cpu_time_us":5408,"apply.run_wall_time_us":1378967,"num_ops":3,"prepare.queue_time_us":228562,"prepare.run_cpu_time_us":36380,"prepare.run_wall_time_us":1836210,"raft.queue_time_us":615311,"replication_time_us":2718461,"spinlock_wait_cycles":10641664,"thread_start_us":1753845,"threads_started":5,"wal-append.queue_time_us":1299373,"wal-append.run_cpu_time_us":40102,"wal-append.run_wall_time_us":2290591}]]}
I20260808 20:14:17.231420 14468 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:17.325178 12141 rpcz_store.cc:276] 0808 20:14:10.014048 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:10.016527 (+ 2479us) service_pool.cc:227] Handling call
0808 20:14:17.165447 (+7148920us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:17.165782 (+ 335us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:17.166163 (+ 381us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:17.166189 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:17.166228 (+ 39us) raft_consensus.cc:1631] Marking committed up to 1
0808 20:14:17.166294 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:17.166351 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:17.166461 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":236287872}
I20260808 20:14:17.335063 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:17.285558 14407 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:17.336410 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.336207 14407 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:17.337780 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.338142 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.338387 14407 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:16.866716 14676 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
W20260808 20:14:17.334236 12140 rpcz_store.cc:269] 0808 20:14:12.241286 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.333642 (+5092356us) service_pool.cc:214] Skipping call since client already timed out
0808 20:14:17.333758 (+ 116us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 20:14:17.318593 12047 raft_consensus.cc:2454] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 1.
W20260808 20:14:17.107764 12144 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1485) took 6584 ms (client timeout 2223 ms). Trace:
W20260808 20:14:17.297806 14428 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 5.123s user 0.181s sys 0.323s
W20260808 20:14:17.108455 12105 connection.cc:232] RPC call timeout handler was delayed by -4.0883307398762554s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19720000000000001s for a response
I20260808 20:14:17.339639 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:17.339962 12105 connection.cc:232] RPC call timeout handler was delayed by -3.7138621010817587s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15580000000000002s for a response
W20260808 20:14:17.340299 12105 connection.cc:232] RPC call timeout handler was delayed by -3.3214621010702103s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19940000000000002s for a response
I20260808 20:14:17.313876 13080 rpcz_store.cc:276] 0808 20:14:10.556858 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:10.557014 (+ 156us) service_pool.cc:227] Handling call
0808 20:14:10.560929 (+ 3915us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:11.117990 (+557061us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:11.118023 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:11.118062 (+ 39us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:11.118092 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:17.151464 (+6033372us) spinlock_profiling.cc:243] Waited 557 ms on lock 0x7b540044cad4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:17.151547 (+ 83us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:17.151832 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1280607616}
I20260808 20:14:17.107755 14683 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:17.342474 12249 raft_consensus.cc:2454] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:17.344732 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.345885 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.346346 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.346909 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:17.339469 12144 rpcz_store.cc:269] 0808 20:14:10.522846 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.107428 (+6584582us) service_pool.cc:214] Skipping call since client already timed out
0808 20:14:17.107592 (+ 164us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 20:14:17.347730 13078 raft_consensus.cc:2454] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:14:17.348109 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:17.349251 14686 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:17.350455 12144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1191) took 6931 ms. Trace:
I20260808 20:14:17.350863 12144 rpcz_store.cc:276] 0808 20:14:10.418686 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.349207 (+6930521us) service_pool.cc:227] Handling call
0808 20:14:17.349664 (+ 457us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:17.349971 (+ 307us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:17.350000 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:17.350089 (+ 89us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:17.350126 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:17.350203 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:17.350355 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:17.339167 14676 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:17.355103 14676 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:14:17.355849 12000 connection.cc:232] RPC call timeout handler was delayed by -6.8315451708622277s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1554s for a response
I20260808 20:14:17.356681 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:17.357481 13080 raft_consensus.cc:2454] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:17.358454 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.358969 12141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 200) took 6813 ms. Trace:
I20260808 20:14:17.359346 12141 rpcz_store.cc:276] 0808 20:14:10.545534 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.357899 (+6812365us) service_pool.cc:227] Handling call
0808 20:14:17.358310 (+ 411us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:17.358646 (+ 336us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:17.358675 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:17.358713 (+ 38us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:17.358747 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:17.358810 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:17.358916 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:17.360234 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:17.313078 12250 rpcz_store.cc:276] 0808 20:14:10.795432 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:10.795553 (+ 121us) service_pool.cc:227] Handling call
0808 20:14:10.991899 (+196346us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:11.122682 (+130783us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:11.122711 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:11.122747 (+ 36us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:14:11.122776 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:17.132878 (+6010102us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b54000bc2d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:17.133310 (+ 432us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:17.133532 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":300346624}
W20260808 20:14:17.305051 13079 rpcz_store.cc:269] 0808 20:14:10.735436 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:10.735606 (+ 170us) service_pool.cc:227] Handling call
0808 20:14:17.119588 (+6383982us) spinlock_profiling.cc:243] Waited 50.7 ms on lock 0x7b540076edd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:17.119787 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":116498688}
W20260808 20:14:17.208348 14426 log.cc:924] Time spent T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 4.366s user 0.025s sys 0.017s
I20260808 20:14:17.359467 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:17.362502 14426 tablet_bootstrap.cc:492] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replaying log segment 1/1 (621B/621B this segment, stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20260808 20:14:17.363039 14426 tablet_bootstrap.cc:492] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 20:14:17.364140 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:17.364874 12045 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:17.364501 14683 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:17.365232 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.365971 12047 raft_consensus.cc:2454] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:14:17.366958 14683 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:17.355881 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:17.367718 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.367991 12036 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:17.217547 12245 raft_consensus.cc:1226] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3. Ops: []
W20260808 20:14:17.327036 12101 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 5.355s user 0.018s sys 0.066s
W20260808 20:14:17.296066 12147 rpcz_store.cc:269] 0808 20:14:10.541697 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.295515 (+6753818us) service_pool.cc:214] Skipping call since client already timed out
0808 20:14:17.295646 (+ 131us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260808 20:14:17.315483 12138 rpcz_store.cc:269] 0808 20:14:10.016247 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:10.079519 (+ 63272us) service_pool.cc:227] Handling call
0808 20:14:17.153621 (+7074102us) spinlock_profiling.cc:243] Waited 63.8 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:17.315053 (+161432us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":146666880}
W20260808 20:14:17.322870 12146 rpcz_store.cc:269] 0808 20:14:11.979985 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.322340 (+5342355us) service_pool.cc:214] Skipping call since client already timed out
0808 20:14:17.322449 (+ 109us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 20:14:17.304613 14684 leader_election.cc:288] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:17.262637 12142 rpcz_store.cc:269] 0808 20:14:11.896375 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.262004 (+5365629us) service_pool.cc:214] Skipping call since client already timed out
0808 20:14:17.262134 (+ 130us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 20:14:17.297361 14682 raft_consensus.cc:504] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:17.333716 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:17.286005 12150 rpcz_store.cc:276] 0808 20:14:10.068026 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:10.444020 (+375994us) service_pool.cc:227] Handling call
0808 20:14:16.934227 (+6490207us) spinlock_profiling.cc:243] Waited 1.6 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:16.934679 (+ 452us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:16.935183 (+ 504us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:16.935206 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:16.935264 (+ 58us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:16.935288 (+ 24us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:16.935409 (+ 121us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:16.935583 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3687680}
W20260808 20:14:17.280303 12241 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1676) took 7293 ms (client timeout 2239 ms). Trace:
I20260808 20:14:17.317296 12137 rpcz_store.cc:276] 0808 20:14:10.015851 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:10.016610 (+ 759us) service_pool.cc:227] Handling call
0808 20:14:17.179681 (+7163071us) spinlock_profiling.cc:243] Waited 64.7 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:17.180023 (+ 342us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:17.180346 (+ 323us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:17.180370 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:17.180426 (+ 56us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:17.180469 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:17.180524 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:17.180634 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":148858112}
I20260808 20:14:17.323179 14681 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:17.316421 11995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 7.168s user 0.042s sys 0.069s
I20260808 20:14:17.341090 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:17.368108 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:17.476100 12038 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
W20260808 20:14:17.490325 14689 log.cc:924] Time spent T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.160s user 0.000s sys 0.001s
W20260808 20:14:17.494765 11994 connection.cc:232] RPC call timeout handler was delayed by -5.1234551859088242s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16080000000000003s for a response
W20260808 20:14:17.495575 11994 connection.cc:232] RPC call timeout handler was delayed by -5.0904575169552118s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17400000000000002s for a response
W20260808 20:14:17.495911 11994 connection.cc:232] RPC call timeout handler was delayed by -5.02174883405678s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1726s for a response
W20260808 20:14:17.496335 11994 connection.cc:232] RPC call timeout handler was delayed by -4.8035960521083325s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1607s for a response
W20260808 20:14:17.497253 11994 connection.cc:232] RPC call timeout handler was delayed by -4.3790385890752077s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16100000000000003s for a response
W20260808 20:14:17.497563 11994 connection.cc:232] RPC call timeout handler was delayed by -4.2358795320615172s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2223s for a response
W20260808 20:14:17.422912 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 134199us
I20260808 20:14:17.334259 11889 catalog_manager.cc:5495] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 1 to 2, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
W20260808 20:14:17.498072 11994 connection.cc:232] RPC call timeout handler was delayed by -3.3150805530603975s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32160000000000005s for a response
I20260808 20:14:17.405421 12152 raft_consensus.cc:2454] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6.
I20260808 20:14:17.537024 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:17.557265 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:17.545352 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.591157 12038 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:17.591827 13078 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:17.441639 12144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 199) took 6831 ms. Trace:
I20260808 20:14:17.592763 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.595945 12043 raft_consensus.cc:2454] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:17.596091 11994 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:17.551244 14682 leader_election.cc:288] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:17.533178 12138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 206) took 5933 ms. Trace:
I20260808 20:14:17.601608 12138 rpcz_store.cc:276] 0808 20:14:11.599706 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.531947 (+5932241us) service_pool.cc:227] Handling call
0808 20:14:17.532420 (+ 473us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:17.532736 (+ 316us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:17.532772 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:17.532834 (+ 62us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:17.532869 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:17.532953 (+ 84us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:17.533090 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:17.528564 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:17.537401 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:17.605387 12150 raft_consensus.cc:2379] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:17.605984 14692 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:17.594911 12146 raft_consensus.cc:2379] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:17.606782 14681 raft_consensus.cc:2781] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:17.594070 12036 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:14:17.607220 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.604668 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:17.552219 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:17.599712 13016 connection.cc:232] RPC call timeout handler was delayed by -5.5455716149881482s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2142s for a response
I20260808 20:14:17.544485 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:17.547677 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 50070.4us
I20260808 20:14:17.552160 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.550587 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.545362 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:17.464234 14428 log.cc:924] Time spent T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.103s user 0.000s sys 0.004s
W20260808 20:14:17.576400 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 5.63827e+06us
I20260808 20:14:17.364183 14426 tablet_bootstrap.cc:492] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:14:17.559134 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.405691 13076 raft_consensus.cc:2454] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:17.346419 14468 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:17.600399 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.557334 12105 leader_election.cc:302] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:17.600713 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:17.601024 12152 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 201) took 7055 ms (client timeout 1920 ms). Trace:
W20260808 20:14:17.491822 12241 rpcz_store.cc:269] 0808 20:14:09.986667 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:09.987231 (+ 564us) service_pool.cc:227] Handling call
0808 20:14:17.279927 (+7292696us) spinlock_profiling.cc:243] Waited 6.1 s on lock 0x7b5400375854. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:17.280233 (+ 306us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14030683008}
I20260808 20:14:17.560272 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:17.623801 12040 raft_consensus.cc:3037] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:17.538547 12245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 217) took 7259 ms. Trace:
I20260808 20:14:17.601147 14575 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:17.526446 12140 raft_consensus.cc:2379] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:17.602533 13073 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 322) took 6862 ms. Trace:
I20260808 20:14:17.552692 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:17.603300 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 85403.5us
I20260808 20:14:17.631052 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:17.367588 14396 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:17.326994 14400 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:17.608407 14684 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:17.493820 14686 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:17.626312 12245 rpcz_store.cc:276] 0808 20:14:10.279389 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:10.546223 (+266834us) service_pool.cc:227] Handling call
0808 20:14:17.217036 (+6670813us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:17.526772 (+309736us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:17.538344 (+ 11572us) spinlock_profiling.cc:243] Waited 6.67 s on lock 0x7b540037cdd0. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d9a09 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:17.538488 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15340730112}
I20260808 20:14:17.609436 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:17.550222 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.630609 14575 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:17.641067 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:17.498303 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 353582us
I20260808 20:14:17.498662 13072 raft_consensus.cc:2454] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:14:17.610846 11889 catalog_manager.cc:5495] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 1 to 2, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260808 20:14:17.497758 12000 connection.cc:232] RPC call timeout handler was delayed by -6.3973509329371154s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16080000000000003s for a response
I20260808 20:14:17.617664 12039 raft_consensus.cc:2454] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
W20260808 20:14:17.618012 12152 rpcz_store.cc:269] 0808 20:14:10.545787 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:12.241618 (+1695831us) service_pool.cc:227] Handling call
0808 20:14:17.590210 (+5348592us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:17.606750 14692 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:17.541558 14407 raft_consensus.cc:482] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:17.548324 12141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 202) took 6982 ms. Trace:
I20260808 20:14:17.552615 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.528112 14533 raft_consensus.cc:482] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:17.625087 13080 raft_consensus.cc:2454] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:17.596994 12144 rpcz_store.cc:276] 0808 20:14:10.545240 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.351882 (+6806642us) service_pool.cc:227] Handling call
0808 20:14:17.352308 (+ 426us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:17.352623 (+ 315us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:17.352653 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:17.352691 (+ 38us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:17.352720 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:17.352776 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:17.376616 (+ 23840us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:17.630570 12046 raft_consensus.cc:2379] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 3: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:17.605283 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.598970 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:17.518286 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.471519 14680 raft_consensus.cc:482] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:17.631891 12045 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:17.541100 13074 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
W20260808 20:14:17.634583 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 77368.1us
I20260808 20:14:17.636137 14396 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:17.342458 12078 debug-util.cc:398] Leaking SignalData structure 0x7b080038d480 after lost signal to thread 12186
W20260808 20:14:17.645490 12000 connection.cc:232] RPC call timeout handler was delayed by -6.3147792217787355s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17400000000000002s for a response
I20260808 20:14:17.637503 14684 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:17.647282 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:17.646960 12147 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:17.650758 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:17.650910 12146 raft_consensus.cc:1226] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3. Ops: []
W20260808 20:14:17.651093 13016 connection.cc:232] RPC call timeout handler was delayed by -4.9332965568173677s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15180000000000002s for a response
I20260808 20:14:17.667364 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:17.648689 14681 raft_consensus.cc:686] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
W20260808 20:14:17.651636 12102 connection.cc:232] RPC call timeout handler was delayed by -4.472317717038095s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1564s for a response
W20260808 20:14:17.654390 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 48127.3us
I20260808 20:14:17.647089 14629 raft_consensus.cc:2781] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:17.657519 12247 raft_consensus.cc:2454] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:14:17.655963 14407 raft_consensus.cc:504] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:17.659049 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.688465 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:17.659783 14533 raft_consensus.cc:504] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:17.691162 12047 raft_consensus.cc:2454] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:17.692528 14533 leader_election.cc:288] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:17.693127 12101 connection.cc:232] RPC call timeout handler was delayed by -5.6210298601072282s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2239s for a response
W20260808 20:14:17.693495 12101 connection.cc:232] RPC call timeout handler was delayed by -5.0000298600643873s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2929s for a response
I20260808 20:14:17.693713 14428 tablet_bootstrap.cc:492] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20260808 20:14:17.693773 12101 connection.cc:232] RPC call timeout handler was delayed by -4.9505118292290717s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15580000000000002s for a response
I20260808 20:14:17.694142 12105 leader_election.cc:302] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:17.684080 14629 raft_consensus.cc:482] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:17.694809 14629 raft_consensus.cc:3037] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:17.695684 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.534049 14344 raft_consensus.cc:348] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:17.654796 14400 raft_consensus.cc:482] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:17.690223 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:17.663053 12000 connection.cc:232] RPC call timeout handler was delayed by -6.2965097739361227s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1726s for a response
I20260808 20:14:17.699357 12043 raft_consensus.cc:2454] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
W20260808 20:14:17.651337 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 153959us
I20260808 20:14:17.696846 14344 raft_consensus.cc:729] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
W20260808 20:14:17.668334 13016 connection.cc:232] RPC call timeout handler was delayed by -4.5064076140988618s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20270000000000002s for a response
I20260808 20:14:17.660820 14672 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:17.700150 12040 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:17.682552 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.682427 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.669783 12152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1200) took 5589 ms. Trace:
I20260808 20:14:17.683166 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.687382 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:17.703115 12038 raft_consensus.cc:3037] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:17.666636 14426 ts_tablet_manager.cc:1397] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 8.898s user 0.068s sys 0.064s
I20260808 20:14:17.685540 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.700459 12140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1198) took 5720 ms. Trace:
W20260808 20:14:17.682734 12205 connection.cc:232] RPC call timeout handler was delayed by -4.6714072509203106s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1547s for a response
I20260808 20:14:17.687336 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:17.689478 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.659372 12137 raft_consensus.cc:1226] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:17.652585 14686 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:17.694918 14428 tablet_bootstrap.cc:492] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20: Bootstrap complete.
I20260808 20:14:17.652767 12041 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:17.712417 12137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 207) took 6112 ms. Trace:
I20260808 20:14:17.712800 12137 rpcz_store.cc:276] 0808 20:14:11.600015 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.612937 (+6012922us) service_pool.cc:227] Handling call
0808 20:14:17.653620 (+ 40683us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:17.712155 (+ 58535us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:17.712355 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:17.713447 12147 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1202) took 5561 ms. Trace:
I20260808 20:14:17.713521 14692 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:17.714567 12137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1203) took 5562 ms. Trace:
I20260808 20:14:17.714793 14683 raft_consensus.cc:2781] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:17.715250 14683 raft_consensus.cc:482] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:17.715622 14683 raft_consensus.cc:3037] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:17.716297 14396 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:17.694231 12101 connection.cc:232] RPC call timeout handler was delayed by -4.8885974131990224s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16410000000000002s for a response
W20260808 20:14:17.717144 12101 connection.cc:232] RPC call timeout handler was delayed by -4.76439741300419s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1779s for a response
I20260808 20:14:17.713932 12147 rpcz_store.cc:276] 0808 20:14:12.151988 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.685312 (+5533324us) service_pool.cc:227] Handling call
0808 20:14:17.705728 (+ 20416us) spinlock_profiling.cc:243] Waited 1.22 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:17.705902 (+ 174us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:17.713158 (+ 7256us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:17.713184 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:17.713222 (+ 38us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:14:17.713248 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:17.713300 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:17.713395 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2807680}
I20260808 20:14:17.658558 12150 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 4
W20260808 20:14:17.699431 12000 connection.cc:232] RPC call timeout handler was delayed by -6.0263302130624652s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1607s for a response
I20260808 20:14:17.668133 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 208) took 5772 ms. Trace:
I20260808 20:14:17.700839 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1195) took 5721 ms. Trace:
I20260808 20:14:17.648778 12141 rpcz_store.cc:276] 0808 20:14:10.546021 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.360359 (+6814338us) service_pool.cc:227] Handling call
0808 20:14:17.360740 (+ 381us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:17.432396 (+ 71656us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:17.464849 (+ 32453us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:17.471679 (+ 6830us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:17.494779 (+ 23100us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:17.499216 (+ 4437us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:17.528481 (+ 29265us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:17.681321 12102 connection.cc:232] RPC call timeout handler was delayed by -4.4267233859281987s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2999s for a response
I20260808 20:14:17.686197 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:17.706192 12205 connection.cc:232] RPC call timeout handler was delayed by -4.52305083302781s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1785s for a response
I20260808 20:14:17.702781 12152 rpcz_store.cc:276] 0808 20:14:12.080922 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.645915 (+5564993us) service_pool.cc:227] Handling call
0808 20:14:17.652233 (+ 6318us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:17.669455 (+ 17222us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:17.669482 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:17.669550 (+ 68us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:14:17.669582 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:17.669638 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:17.669734 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:17.723166 12102 connection.cc:232] RPC call timeout handler was delayed by -4.4030177171807736s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16410000000000002s for a response
I20260808 20:14:17.723531 12244 raft_consensus.cc:2354] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4.
I20260808 20:14:17.701851 12138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1192) took 5718 ms. Trace:
I20260808 20:14:17.710820 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.711015 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:17.720029 12146 rpcz_store.cc:276] 0808 20:14:11.895833 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.607059 (+5711226us) service_pool.cc:227] Handling call
0808 20:14:17.644319 (+ 37260us) spinlock_profiling.cc:243] Waited 879 us on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:17.650743 (+ 6424us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:17.667905 (+ 17162us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:17.668079 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2021120}
I20260808 20:14:17.726337 13080 raft_consensus.cc:2454] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:14:17.711783 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:17.717481 12101 connection.cc:232] RPC call timeout handler was delayed by -4.6995974131859839s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18510000000000001s for a response
W20260808 20:14:17.719954 12000 connection.cc:232] RPC call timeout handler was delayed by -5.8762112229596823s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16100000000000003s for a response
W20260808 20:14:17.708139 12078 debug-util.cc:398] Leaking SignalData structure 0x7b080039e700 after lost signal to thread 12287
W20260808 20:14:17.727010 12101 connection.cc:232] RPC call timeout handler was delayed by -4.5141974131111056s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2057s for a response
I20260808 20:14:17.722407 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.720121 12142 rpcz_store.cc:276] 0808 20:14:11.979742 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.644106 (+5664364us) service_pool.cc:227] Handling call
0808 20:14:17.661504 (+ 17398us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:17.686101 (+ 24597us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:17.686130 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:17.686170 (+ 40us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:17.690808 (+ 4638us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:17.700683 (+ 9875us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:17.700787 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:17.723572 12102 connection.cc:232] RPC call timeout handler was delayed by -4.2788177169859409s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1779s for a response
I20260808 20:14:17.704702 12140 rpcz_store.cc:276] 0808 20:14:11.980527 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.644433 (+5663906us) service_pool.cc:227] Handling call
0808 20:14:17.660859 (+ 16426us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:17.698859 (+ 38000us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:17.700160 (+ 1301us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:17.700204 (+ 44us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:17.700236 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:17.700292 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:17.700405 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:17.725340 12205 connection.cc:232] RPC call timeout handler was delayed by -4.06122967787087s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1892s for a response
I20260808 20:14:17.709517 14672 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:17.720417 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:17.702764 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.700721 14680 raft_consensus.cc:504] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:17.727409 12101 connection.cc:232] RPC call timeout handler was delayed by -4.4673974129837006s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2109s for a response
W20260808 20:14:17.762029 12205 connection.cc:232] RPC call timeout handler was delayed by -3.9610412439797074s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15700000000000003s for a response
W20260808 20:14:17.761534 12102 connection.cc:232] RPC call timeout handler was delayed by -4.2707177170086652s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17880000000000001s for a response
W20260808 20:14:17.727090 12000 connection.cc:232] RPC call timeout handler was delayed by -5.4947516920510679s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2223s for a response
I20260808 20:14:17.761639 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:17.717372 14684 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:17.629310 13073 rpcz_store.cc:276] 0808 20:14:10.740383 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:10.745751 (+ 5368us) service_pool.cc:227] Handling call
0808 20:14:17.203173 (+6457422us) spinlock_profiling.cc:243] Waited 18.3 ms on lock 0x7b5800005198. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:17.203528 (+ 355us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:17.256131 (+ 52603us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:17.256172 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:17.256754 (+ 582us) log.cc:842] Serialized 26 byte log entry
0808 20:14:17.495445 (+238691us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:17.496973 (+ 1528us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:17.546493 (+ 49520us) spinlock_profiling.cc:243] Waited 52.1 ms on lock 0x7b5400030554. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:17.546567 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:17.593901 (+ 47334us) spinlock_profiling.cc:243] Waited 20 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:17.602098 (+ 8197us) raft_consensus.cc:1671] finished
0808 20:14:17.602130 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:17.602328 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":208027008,"thread_start_us":165663,"threads_started":1,"wal-append.queue_time_us":294635}
I20260808 20:14:17.789819 14681 consensus_queue.cc:237] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:17.790731 12041 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:17.790900 14686 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:17.764359 12206 connection.cc:232] RPC call timeout handler was delayed by -3.9680792652070522s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1547s for a response
I20260808 20:14:17.761193 12250 raft_consensus.cc:2454] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
W20260808 20:14:17.786854 12205 connection.cc:232] RPC call timeout handler was delayed by -3.7406711149960756s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33210000000000006s for a response
W20260808 20:14:17.787062 12102 connection.cc:232] RPC call timeout handler was delayed by -4.2692233859561384s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3174s for a response
W20260808 20:14:17.727473 12078 debug-util.cc:398] Leaking SignalData structure 0x7b0800107740 after lost signal to thread 12288
I20260808 20:14:17.765865 12105 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:17.796408 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.787488 12249 raft_consensus.cc:2362] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3. Re-sending same reply.
I20260808 20:14:17.797112 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.797760 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.723872 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:17.788338 12101 connection.cc:232] RPC call timeout handler was delayed by -4.4039774800185114s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21480000000000002s for a response
I20260808 20:14:17.789654 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.785215 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:17.791836 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:17.792106 12205 connection.cc:232] RPC call timeout handler was delayed by -3.4780927170068026s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19390000000000002s for a response
W20260808 20:14:17.792151 12102 connection.cc:232] RPC call timeout handler was delayed by -4.22872338606976s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3219s for a response
W20260808 20:14:17.792546 12206 connection.cc:232] RPC call timeout handler was delayed by -3.9113792651332915s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16100000000000003s for a response
W20260808 20:14:17.788298 12000 connection.cc:232] RPC call timeout handler was delayed by -4.9211302129551768s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.28350000000000003s for a response
I20260808 20:14:17.796916 12046 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:14:17.798833 12101 connection.cc:232] RPC call timeout handler was delayed by -4.21820665593259s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3174s for a response
I20260808 20:14:17.799147 12147 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1204) took 5646 ms. Trace:
W20260808 20:14:17.799468 12102 connection.cc:232] RPC call timeout handler was delayed by -4.2140177171677351s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18510000000000001s for a response
W20260808 20:14:17.763561 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 52812.3us
W20260808 20:14:17.799961 12206 connection.cc:232] RPC call timeout handler was delayed by -3.7538792651612312s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1785s for a response
W20260808 20:14:17.800275 12101 connection.cc:232] RPC call timeout handler was delayed by -4.1777066560462117s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3219s for a response
I20260808 20:14:17.799826 12147 rpcz_store.cc:276] 0808 20:14:12.152675 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.718888 (+5566213us) service_pool.cc:227] Handling call
0808 20:14:17.787067 (+ 68179us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:17.798863 (+ 11796us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:17.798890 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:17.798927 (+ 37us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:17.798957 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:17.799010 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:17.799097 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:17.800784 12206 connection.cc:232] RPC call timeout handler was delayed by -3.7241792650893331s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18180000000000002s for a response
I20260808 20:14:17.722087 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:17.801321 12000 connection.cc:232] RPC call timeout handler was delayed by -4.5400075528305024s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32160000000000005s for a response
W20260808 20:14:17.800033 12102 connection.cc:232] RPC call timeout handler was delayed by -4.1051177170593292s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19720000000000001s for a response
W20260808 20:14:17.801713 12206 connection.cc:232] RPC call timeout handler was delayed by -3.6575792650692165s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1892s for a response
I20260808 20:14:17.800566 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
W20260808 20:14:17.799383 12205 connection.cc:232] RPC call timeout handler was delayed by -3.0603642358910292s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2574s for a response
I20260808 20:14:17.801208 12039 raft_consensus.cc:2379] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
W20260808 20:14:17.806370 12206 connection.cc:232] RPC call timeout handler was delayed by -2.9069792651571333s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2726s for a response
W20260808 20:14:17.806931 12206 connection.cc:232] RPC call timeout handler was delayed by -2.7242792651522905s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2929s for a response
I20260808 20:14:17.804533 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:17.807277 12206 connection.cc:232] RPC call timeout handler was delayed by -2.576679265126586s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3093s for a response
W20260808 20:14:17.808410 12206 connection.cc:232] RPC call timeout handler was delayed by -2.371479265158996s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33210000000000006s for a response
W20260808 20:14:17.806250 12102 connection.cc:232] RPC call timeout handler was delayed by -4.0991233859676868s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33630000000000004s for a response
I20260808 20:14:17.727692 14407 leader_election.cc:288] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:17.809115 12102 connection.cc:232] RPC call timeout handler was delayed by -3.9818177169654518s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2109s for a response
I20260808 20:14:17.799140 13077 raft_consensus.cc:2454] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
W20260808 20:14:17.810758 12102 connection.cc:232] RPC call timeout handler was delayed by -3.9467177169863135s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21480000000000002s for a response
W20260808 20:14:17.811264 12102 connection.cc:232] RPC call timeout handler was delayed by -3.7461929060518742s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1668s for a response
W20260808 20:14:17.811546 12102 connection.cc:232] RPC call timeout handler was delayed by -3.5588746210560203s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17830000000000001s for a response
W20260808 20:14:17.811811 12102 connection.cc:232] RPC call timeout handler was delayed by -3.468092906055972s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19770000000000001s for a response
W20260808 20:14:17.812058 12102 connection.cc:232] RPC call timeout handler was delayed by -3.3424428580328822s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2021s for a response
W20260808 20:14:17.812304 12102 connection.cc:232] RPC call timeout handler was delayed by -3.3267929060384631s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2134s for a response
I20260808 20:14:17.720172 14400 raft_consensus.cc:504] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:17.788022 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:17.812577 12102 connection.cc:232] RPC call timeout handler was delayed by -3.2820139368996024s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1892s for a response
I20260808 20:14:17.821283 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.821421 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.822372 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.823184 12251 raft_consensus.cc:3037] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:17.825867 13080 raft_consensus.cc:1226] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:17.825682 14606 raft_consensus.cc:2726] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:17.824528 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.828444 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.720304 14344 consensus_queue.cc:260] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:17.828629 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.815830 11994 leader_election.cc:302] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:17.830318 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:17.829579 12043 raft_consensus.cc:1226] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:17.833897 12249 raft_consensus.cc:2379] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:17.822968 14428 ts_tablet_manager.cc:1397] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent bootstrapping tablet: real 9.000s user 0.215s sys 0.360s
I20260808 20:14:17.717911 12137 rpcz_store.cc:276] 0808 20:14:12.152290 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.713622 (+5561332us) service_pool.cc:227] Handling call
0808 20:14:17.713991 (+ 369us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:17.714236 (+ 245us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:17.714270 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:17.714308 (+ 38us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:14:17.714336 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:17.714387 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:17.714499 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:17.841110 14344 ts_tablet_manager.cc:1428] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.510s user 0.015s sys 0.004s
W20260808 20:14:17.841463 14344 ts_tablet_manager.cc:1460] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20: Tablet startup took 510ms
W20260808 20:14:17.842435 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 137955us
I20260808 20:14:17.843626 12105 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:17.844389 14606 raft_consensus.cc:2781] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:17.844769 14606 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:17.845317 14606 raft_consensus.cc:3037] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:14:17.841791 14344 ts_tablet_manager.cc:1462] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20: Trace:
0808 20:14:03.362391 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:03.545016 (+182625us) service_pool.cc:227] Handling call
0808 20:14:04.209369 (+664353us) ts_tablet_manager.cc:928] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
0808 20:14:06.659619 (+2450250us) tablet_copy_client.cc:361] Tablet copy session begun
0808 20:14:07.250873 (+591254us) tablet_metadata.cc:641] Metadata flushed
0808 20:14:07.318391 (+ 67518us) tablet_copy_client.cc:490] Wrote new tablet metadata
0808 20:14:07.640861 (+322470us) tablet_copy_client.cc:411] Wrote new consensus metadata
0808 20:14:07.659049 (+ 18188us) tablet_replica.cc:205] Creating consensus instance
0808 20:14:07.659498 (+ 449us) ts_tablet_manager.cc:983] Replica registered.
0808 20:14:07.659605 (+ 107us) inbound_call.cc:173] Queueing success response
0808 20:14:08.193700 (+534095us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0808 20:14:17.330657 (+9136957us) ts_tablet_manager.cc:1429] Starting tablet replica
0808 20:14:17.331398 (+ 741us) tablet_replica.cc:259] Starting instrumentation
0808 20:14:17.332126 (+ 728us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":242,"raft.queue_time_us":124,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":100,"tablet-copy.queue_time_us":663491,"tablet-download-pool.queue_time_us":631,"tablet-download-pool.run_cpu_time_us":1558,"tablet-download-pool.run_wall_time_us":57557,"thread_start_us":1082593,"threads_started":2}
I20260808 20:14:17.844485 12105 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:17.848192 12046 raft_consensus.cc:2454] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
W20260808 20:14:17.854390 12078 debug-util.cc:398] Leaking SignalData structure 0x7b08002adde0 after lost signal to thread 13118
W20260808 20:14:17.854714 12078 debug-util.cc:398] Leaking SignalData structure 0x7b0800103c40 after lost signal to thread 13119
W20260808 20:14:17.855010 12078 debug-util.cc:398] Leaking SignalData structure 0x7b08002dd8e0 after lost signal to thread 14345
W20260808 20:14:17.856158 12078 debug-util.cc:398] Leaking SignalData structure 0x7b08003976c0 after lost signal to thread 14578
W20260808 20:14:17.856463 12078 debug-util.cc:398] Leaking SignalData structure 0x7b0800397700 after lost signal to thread 14591
I20260808 20:14:17.857914 14400 leader_election.cc:288] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:17.859843 14400 raft_consensus.cc:2781] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:17.860256 14400 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:17.861480 14680 leader_election.cc:288] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:17.726788 12138 rpcz_store.cc:276] 0808 20:14:11.978859 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.607724 (+5628865us) service_pool.cc:227] Handling call
0808 20:14:17.682385 (+ 74661us) spinlock_profiling.cc:243] Waited 2.95 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:17.695109 (+ 12724us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:17.695385 (+ 276us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:17.695418 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:17.695472 (+ 54us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:17.695505 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:17.695564 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:17.697108 (+ 1544us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6793472}
I20260808 20:14:17.860692 14400 raft_consensus.cc:3037] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:17.867425 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:17.866490 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 61837.1us
I20260808 20:14:17.866976 14680 raft_consensus.cc:2781] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:17.868788 14680 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:17.868940 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.869287 14680 raft_consensus.cc:3037] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:17.869709 14575 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:17.870306 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:17.875276 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.877995 12038 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:17.875924 12046 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
W20260808 20:14:17.856810 12078 debug-util.cc:398] Leaking SignalData structure 0x7b0800397800 after lost signal to thread 14604
W20260808 20:14:17.880177 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 48803.6us
I20260808 20:14:17.829690 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.830140 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:17.885156 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 61319.4us
I20260808 20:14:17.887025 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.887605 12040 raft_consensus.cc:2454] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:17.887557 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:17.885637 12150 raft_consensus.cc:2454] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:17.843564 14426 raft_consensus.cc:348] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:17.891283 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:17.891616 14426 raft_consensus.cc:729] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:14:17.888410 13079 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:17.905598 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.895313 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.923970 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:17.924252 14432 consensus_queue.cc:1046] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:17.926335 14432 raft_consensus.cc:3032] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:17.926445 14672 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:17.927115 14432 raft_consensus.cc:729] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:17.928241 14432 consensus_queue.cc:260] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:17.875377 14629 raft_consensus.cc:504] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:17.929528 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:17.929724 12000 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:17.929482 14432 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:17.930229 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:17.930701 12000 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:17.931370 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:17.932540 14575 raft_consensus.cc:2741] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260808 20:14:17.933153 14575 raft_consensus.cc:2781] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:17.933538 14575 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:17.933897 14575 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:17.938263 14629 leader_election.cc:288] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:17.946237 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:17.947270 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.947436 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.948097 12036 raft_consensus.cc:2354] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4.
I20260808 20:14:17.950058 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.950542 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.951210 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.951980 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.879467 14606 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:17.953136 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 45192.1us
I20260808 20:14:17.954913 14414 consensus_queue.cc:1046] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:17.956319 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:17.957863 14630 raft_consensus.cc:3032] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:17.958253 14630 raft_consensus.cc:729] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:17.958998 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:17.959264 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 43490.3us
I20260808 20:14:17.959095 14630 consensus_queue.cc:260] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:17.959578 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:17.960440 14630 raft_consensus.cc:3037] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:17.953330 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.962951 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:17.967046 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 71574.3us
I20260808 20:14:17.967417 12038 raft_consensus.cc:2379] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 3: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:17.969477 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.970126 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.970150 12039 raft_consensus.cc:2454] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:17.973416 12000 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:17.974561 12000 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:17.975780 12000 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:17.976500 14686 raft_consensus.cc:2781] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:17.976887 14686 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:17.977281 14686 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:17.979386 14676 consensus_queue.cc:1046] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:17.980850 14676 raft_consensus.cc:3032] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:17.981227 14676 raft_consensus.cc:729] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:17.982240 14676 consensus_queue.cc:260] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:17.983558 14676 raft_consensus.cc:3037] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:17.973441 12247 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 20:14:17.986625 14606 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:17.988094 14428 raft_consensus.cc:348] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:17.988994 14428 raft_consensus.cc:729] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Initialized, Role: LEARNER
I20260808 20:14:17.989234 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.989993 12036 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:17.990193 14699 raft_consensus.cc:2781] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:17.981093 14684 raft_consensus.cc:2781] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:17.990738 14699 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:17.990800 14684 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:17.991281 14684 raft_consensus.cc:3037] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:17.992110 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:17.993477 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:17.892340 14426 consensus_queue.cc:260] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:17.995735 14426 ts_tablet_manager.cc:1428] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.287s user 0.002s sys 0.006s
I20260808 20:14:18.008374 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:18.009141 14699 raft_consensus.cc:3037] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:17.971640 14683 raft_consensus.cc:504] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:18.017478 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:18.023273 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 100794us
I20260808 20:14:18.033246 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:18.023903 14575 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:18.057346 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:18.057530 14428 consensus_queue.cc:260] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:18.057379 12144 raft_consensus.cc:1226] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:18.059508 12144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1201) took 5908 ms. Trace:
I20260808 20:14:18.059844 12144 rpcz_store.cc:276] 0808 20:14:12.151599 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.665542 (+5513943us) service_pool.cc:227] Handling call
0808 20:14:17.721354 (+ 55812us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:18.058643 (+337289us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:18.059241 (+ 598us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b54003fff54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:18.059430 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":677585152}
I20260808 20:14:18.062237 12038 raft_consensus.cc:2362] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3. Re-sending same reply.
I20260808 20:14:18.063941 12251 raft_consensus.cc:2454] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:18.057284 12101 leader_election.cc:302] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:18.074698 14686 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:18.082690 14686 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:18.086153 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:18.086400 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:18.088560 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:18.088613 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:18.091506 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 247055us
I20260808 20:14:18.089346 12047 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:18.096282 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:18.119625 12105 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:18.073369 13073 raft_consensus.cc:2454] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:18.134235 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:18.120463 14669 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.036s sys 0.040s
I20260808 20:14:18.083559 14400 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:18.140478 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 247627us
I20260808 20:14:18.140069 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:18.142094 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:18.142267 12251 raft_consensus.cc:1261] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260808 20:14:18.143322 12251 pending_rounds.cc:76] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a: Aborting all ops after (but not including) 5
I20260808 20:14:18.092355 14684 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:18.143540 12251 pending_rounds.cc:97] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260808 20:14:18.146551 12251 raft_consensus.cc:2866] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 20:14:18.142351 14669 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.594s user 0.036s sys 0.041s
I20260808 20:14:18.138486 14414 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.017s
I20260808 20:14:18.153223 14606 raft_consensus.cc:2781] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:18.153888 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:18.154346 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 231156us
I20260808 20:14:18.136564 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:18.160665 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 64406.8us
I20260808 20:14:18.148588 14683 leader_election.cc:288] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:18.164074 14676 consensus_queue.cc:1035] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 20:14:18.165992 12040 raft_consensus.cc:2379] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 3: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:18.171802 12105 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:18.170862 14704 raft_consensus.cc:2781] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:18.173471 14704 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:18.173823 14704 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:18.157409 14400 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:18.184857 12048 raft_consensus.cc:1261] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 20:14:18.185632 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:18.186409 12045 raft_consensus.cc:2354] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 4. Current term is 5.
I20260808 20:14:18.188035 14575 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:18.188122 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.189038 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.189652 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.190291 12150 raft_consensus.cc:2379] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:18.189281 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.193184 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.194031 12138 raft_consensus.cc:2379] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 2: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:18.196534 13079 raft_consensus.cc:1261] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:14:18.201753 12061 tablet_copy_service.cc:342] P 4cd46f13bceb4c3c9ef71e755e17b995: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-eec183c460ad4f1b85ad1407c53f2eab received from {username='slave'} at 127.0.0.1:49226
I20260808 20:14:18.202277 12061 tablet_copy_service.cc:433] P 4cd46f13bceb4c3c9ef71e755e17b995: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-eec183c460ad4f1b85ad1407c53f2eab on tablet eec183c460ad4f1b85ad1407c53f2eab with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:18.198446 14684 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:18.203935 12001 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:18.171075 14606 raft_consensus.cc:686] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:18.204871 14446 consensus_queue.cc:1035] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:14:18.205880 14606 consensus_queue.cc:237] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:18.200724 12137 raft_consensus.cc:2379] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
W20260808 20:14:18.209146 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 47501.9us
I20260808 20:14:18.210747 13015 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:18.215896 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:18.217036 13079 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:18.217648 13015 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:18.222327 12039 raft_consensus.cc:1261] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 20:14:18.225998 12105 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:18.226171 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:18.226917 12001 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:18.227257 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.228084 12150 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:18.227238 12040 raft_consensus.cc:2362] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3. Re-sending same reply.
I20260808 20:14:18.220723 14400 raft_consensus.cc:2781] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:18.225464 13015 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:18.230729 14575 raft_consensus.cc:2781] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:18.221541 14676 raft_consensus.cc:2781] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:18.186414 14699 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:18.221818 14414 raft_consensus.cc:2726] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:18.238034 12105 leader_election.cc:302] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:18.220112 14684 raft_consensus.cc:2781] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:18.235805 14575 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:18.239306 14575 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:18.239341 13015 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:18.240577 13015 leader_election.cc:302] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:18.236145 14629 raft_consensus.cc:2781] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:18.243206 14629 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:18.243743 14414 raft_consensus.cc:2726] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:18.237269 14400 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:18.244928 14400 raft_consensus.cc:3037] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:18.245703 12101 leader_election.cc:302] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:18.237879 14676 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:18.246794 14676 raft_consensus.cc:3037] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:18.243693 14629 raft_consensus.cc:3037] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:18.249974 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.249717 14575 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:18.251786 14699 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:18.252696 14575 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:18.253262 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 76709.9us
I20260808 20:14:18.242413 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:18.253612 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:18.253945 12040 raft_consensus.cc:3037] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:18.254252 13079 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:18.253916 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.258395 12206 leader_election.cc:302] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:18.249018 14355 raft_consensus.cc:2781] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:18.260021 12105 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:18.261046 13015 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:18.243186 14468 raft_consensus.cc:2781] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:18.263695 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:18.271891 12045 raft_consensus.cc:2379] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 3: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:18.264396 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:18.264513 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:18.265127 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:18.265760 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:18.260179 14683 raft_consensus.cc:2781] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:18.269502 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:18.261843 14355 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:18.242784 14684 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:14:18.306319 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 69092.8us
I20260808 20:14:18.269389 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:18.309774 13079 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:18.312623 12001 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:18.301046 12036 raft_consensus.cc:2379] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:18.271353 14706 raft_consensus.cc:2726] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:18.315616 12040 raft_consensus.cc:2454] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:18.315059 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.319865 12205 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:18.320950 14722 raft_consensus.cc:2781] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:18.321183 12205 leader_election.cc:302] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:18.249718 14704 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:18.322069 12205 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:18.323009 14704 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:18.323976 12205 leader_election.cc:302] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:18.283380 14444 raft_consensus.cc:2726] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:18.325345 14397 raft_consensus.cc:2781] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:18.325843 14397 raft_consensus.cc:686] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:18.326237 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.326725 14397 consensus_queue.cc:237] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:18.327622 14686 raft_consensus.cc:2781] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:18.328133 14686 raft_consensus.cc:686] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:18.321312 14722 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:18.329558 14722 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:18.331489 14629 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:18.324808 14407 raft_consensus.cc:2741] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260808 20:14:18.333272 12206 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:18.334368 14407 raft_consensus.cc:2726] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:18.316648 14606 raft_consensus.cc:2781] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:18.335497 14606 raft_consensus.cc:686] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:18.312778 14414 raft_consensus.cc:2741] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260808 20:14:18.336365 14606 consensus_queue.cc:237] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:18.339581 14428 ts_tablet_manager.cc:1428] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent starting tablet: real 0.501s user 0.014s sys 0.005s
W20260808 20:14:18.339893 14428 ts_tablet_manager.cc:1460] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20: Tablet startup took 502ms
I20260808 20:14:18.305444 14355 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:18.305606 14684 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:18.305823 14468 raft_consensus.cc:686] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:18.257967 14680 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:18.301793 14683 raft_consensus.cc:482] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:18.325728 14444 raft_consensus.cc:2726] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:18.353869 12101 leader_election.cc:302] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:18.352170 12146 raft_consensus.cc:2454] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:14:18.351992 11994 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:18.327283 12144 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:18.359779 14414 consensus_queue.cc:1035] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 20:14:18.364109 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.368474 14575 raft_consensus.cc:2781] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:18.368868 14575 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:18.369473 14575 raft_consensus.cc:3037] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:18.370124 12041 raft_consensus.cc:2379] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 3: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
W20260808 20:14:18.373034 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 113954us
I20260808 20:14:18.363409 14683 raft_consensus.cc:3037] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:18.383311 12163 tablet_copy_service.cc:342] P 78b9263889cd4f1582578a3aa4f644d9: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-2607c6f53fc243d2abb6b2f74b6e2aa6 received from {username='slave'} at 127.0.0.1:46680
I20260808 20:14:18.388684 12163 tablet_copy_service.cc:433] P 78b9263889cd4f1582578a3aa4f644d9: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-2607c6f53fc243d2abb6b2f74b6e2aa6 on tablet 2607c6f53fc243d2abb6b2f74b6e2aa6 with peer e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:18.340175 14428 ts_tablet_manager.cc:1462] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20: Trace:
0808 20:14:05.877326 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:05.881300 (+ 3974us) service_pool.cc:227] Handling call
0808 20:14:06.043489 (+162189us) ts_tablet_manager.cc:928] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20: Initiating tablet copy from peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
0808 20:14:06.828259 (+784770us) tablet_copy_client.cc:361] Tablet copy session begun
0808 20:14:07.559211 (+730952us) tablet_metadata.cc:641] Metadata flushed
0808 20:14:07.562230 (+ 3019us) tablet_copy_client.cc:490] Wrote new tablet metadata
0808 20:14:07.842220 (+279990us) tablet_copy_client.cc:411] Wrote new consensus metadata
0808 20:14:07.857587 (+ 15367us) tablet_replica.cc:205] Creating consensus instance
0808 20:14:07.904615 (+ 47028us) ts_tablet_manager.cc:983] Replica registered.
0808 20:14:07.909009 (+ 4394us) inbound_call.cc:173] Queueing success response
0808 20:14:08.823234 (+914225us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0808 20:14:17.872103 (+9048869us) ts_tablet_manager.cc:1429] Starting tablet replica
0808 20:14:17.941704 (+ 69601us) tablet_replica.cc:259] Starting instrumentation
0808 20:14:17.942410 (+ 706us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":120439,"mutex_wait_us":31515,"raft.queue_time_us":25273,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":111,"tablet-copy.queue_time_us":78454,"tablet-download-pool.queue_time_us":19640,"tablet-download-pool.run_cpu_time_us":2181,"tablet-download-pool.run_wall_time_us":351674,"thread_start_us":171820,"threads_started":2}
I20260808 20:14:18.383515 12146 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 20:14:18.390558 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.384066 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.391194 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.391798 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.392521 12142 raft_consensus.cc:2379] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:18.385562 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.386899 14407 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:18.386395 14676 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:18.383908 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.362591 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:18.407891 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.408560 12150 raft_consensus.cc:2454] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:18.409787 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:18.411890 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 87966.4us
I20260808 20:14:18.412763 12205 leader_election.cc:302] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:18.414304 12205 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:18.415611 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:18.416469 12248 raft_consensus.cc:2396] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 20:14:18.416294 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:18.417165 12243 raft_consensus.cc:2379] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:18.404613 14629 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:18.417903 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:18.356858 14686 consensus_queue.cc:237] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:18.419085 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:18.419939 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:18.420698 12254 raft_consensus.cc:2379] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:18.421891 14400 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:18.423974 14400 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:18.424685 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.425194 12137 raft_consensus.cc:2379] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:18.425552 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:18.426630 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:18.427976 12101 leader_election.cc:398] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Vote denied by peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) with higher term. Message: Invalid argument: T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4.
I20260808 20:14:18.428275 12101 leader_election.cc:401] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260808 20:14:18.428936 12101 leader_election.cc:302] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:18.429155 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.429803 12101 leader_election.cc:302] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:18.425254 14407 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:18.430560 14706 raft_consensus.cc:2781] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:18.430619 14468 consensus_queue.cc:237] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:18.431210 14706 raft_consensus.cc:482] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:18.431598 14706 raft_consensus.cc:3037] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:18.432245 14407 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:18.433974 14676 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:18.437186 12241 raft_consensus.cc:2379] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:18.429692 14629 raft_consensus.cc:2726] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) with higher term. Message: Invalid argument: T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4.
I20260808 20:14:18.437963 14629 raft_consensus.cc:2781] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:18.438532 14629 raft_consensus.cc:686] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:18.405620 14732 raft_consensus.cc:2781] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:18.439558 14629 consensus_queue.cc:237] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:18.440065 14732 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:18.409780 12141 raft_consensus.cc:2362] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6. Re-sending same reply.
I20260808 20:14:18.441130 14732 raft_consensus.cc:3037] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:18.446527 14706 raft_consensus.cc:504] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:18.438561 12101 leader_election.cc:302] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:18.453459 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 103660us
I20260808 20:14:18.455201 12205 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:18.456158 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:18.454960 14444 raft_consensus.cc:2726] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:18.441047 14680 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:18.457540 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:18.457819 12251 raft_consensus.cc:2379] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:18.459949 14355 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:18.464538 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.465265 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:18.466274 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:18.467528 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.457428 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.430171 14722 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:18.469869 14722 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:18.471524 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:18.473026 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:18.481066 12205 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:18.483255 14444 raft_consensus.cc:2781] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:18.483669 14444 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:18.484045 14444 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:18.466729 14432 raft_consensus.cc:2781] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:18.485415 14432 raft_consensus.cc:686] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:18.486271 14432 consensus_queue.cc:237] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:18.486713 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.487926 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:18.488647 13075 raft_consensus.cc:2354] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4.
I20260808 20:14:18.490527 13072 raft_consensus.cc:1203] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 20:14:18.491818 14575 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:18.495750 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:18.495748 12205 leader_election.cc:302] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:18.499210 14722 raft_consensus.cc:2726] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:18.471729 14400 raft_consensus.cc:2726] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:18.527709 12101 leader_election.cc:302] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:18.528146 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:18.528744 12038 raft_consensus.cc:3037] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:18.484143 14355 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:18.532514 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:18.534090 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.530264 14669 raft_consensus.cc:2781] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:18.591807 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:18.592280 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:18.589118 12244 raft_consensus.cc:1226] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:18.590553 12150 raft_consensus.cc:2454] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:18.594969 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.595328 13079 raft_consensus.cc:1226] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:18.592449 12039 raft_consensus.cc:2362] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4. Re-sending same reply.
I20260808 20:14:18.535382 12061 tablet_copy_service.cc:342] P 4cd46f13bceb4c3c9ef71e755e17b995: Request end of tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-21889249378248049ef043f0588cebc9 received from {username='slave'} at 127.0.0.1:49226
I20260808 20:14:18.533329 14706 leader_election.cc:288] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:18.539736 14400 raft_consensus.cc:2781] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:18.601156 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:18.603233 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:18.603719 12147 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:18.535229 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:18.603925 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:18.601692 12150 raft_consensus.cc:1226] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:18.602509 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.604954 12146 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:18.605576 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.525585 14683 raft_consensus.cc:504] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:18.607153 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:18.593186 14575 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:18.609223 14683 leader_election.cc:288] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:18.609966 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:18.611887 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:18.612129 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:18.613708 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:18.615682 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:18.616765 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:18.617370 12039 raft_consensus.cc:2379] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:18.591929 14669 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:18.625053 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:18.632164 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:18.634361 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:18.634452 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:18.632040 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.612994 12061 tablet_copy_service.cc:433] P 4cd46f13bceb4c3c9ef71e755e17b995: ending tablet copy session e5bd9a4616ce4acfa20dc7a002984a20-21889249378248049ef043f0588cebc9 on tablet 21889249378248049ef043f0588cebc9 with peer e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:18.671562 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:18.672230 12047 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:18.673087 12150 raft_consensus.cc:2379] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:18.607165 12036 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:18.673878 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:18.670975 12250 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:14:18.678205 13077 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEARNER]: Advancing to term 4
I20260808 20:14:18.681416 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:18.681634 12147 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:18.682421 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:18.682986 12039 raft_consensus.cc:2362] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6. Re-sending same reply.
I20260808 20:14:18.676682 12038 raft_consensus.cc:2454] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:18.630875 14669 raft_consensus.cc:3037] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:18.684341 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:18.674743 14716 log.cc:924] Time spent T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260808 20:14:18.754153 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:18.631178 14400 raft_consensus.cc:686] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:18.763962 12047 raft_consensus.cc:1226] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:18.769433 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:18.684324 12146 raft_consensus.cc:2379] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:18.674288 14684 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:18.972303 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:18.809787 12038 raft_consensus.cc:1226] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:19.057394 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:18.928200 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:18.967571 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:18.949528 11994 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:18.770535 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:19.080813 12105 leader_election.cc:398] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Vote denied by peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901) with higher term. Message: Invalid argument: T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4.
I20260808 20:14:19.100919 12250 raft_consensus.cc:3037] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:14:18.912784 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 228064us
I20260808 20:14:19.062600 13077 raft_consensus.cc:1261] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEARNER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 20:14:19.112644 12105 leader_election.cc:401] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260808 20:14:18.927978 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:18.899354 13015 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:18.966437 13080 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.011s sys 0.005s
W20260808 20:14:19.115808 13080 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.767s user 0.017s sys 0.019s
W20260808 20:14:19.070008 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 470212us
I20260808 20:14:19.103448 12147 raft_consensus.cc:3037] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 5
I20260808 20:14:19.055893 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:19.009670 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:18.972852 12205 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:19.110145 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 160701us
W20260808 20:14:18.920759 14745 log.cc:924] Time spent T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.152s user 0.001s sys 0.000s
W20260808 20:14:19.111057 12101 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.003s sys 0.008s
W20260808 20:14:19.061800 14748 log.cc:924] Time spent T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.377s user 0.001s sys 0.000s
I20260808 20:14:19.061707 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
W20260808 20:14:19.108441 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 199272us
I20260808 20:14:19.127303 13079 raft_consensus.cc:3037] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:19.117399 12206 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:19.105270 12249 raft_consensus.cc:3037] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 3
I20260808 20:14:19.119760 12142 raft_consensus.cc:2354] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for earlier term 5. Current term is 6.
W20260808 20:14:19.112496 14749 log.cc:924] Time spent T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.193s user 0.001s sys 0.000s
I20260808 20:14:19.121411 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:19.122663 11994 leader_election.cc:302] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:19.123662 12101 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.589s user 0.003s sys 0.008s
I20260808 20:14:19.120234 13080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53290 (request call id 435) took 1158 ms. Trace:
I20260808 20:14:19.110736 14575 raft_consensus.cc:2726] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:19.158030 12146 raft_consensus.cc:2379] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:18.972617 14444 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:19.124934 12045 raft_consensus.cc:1226] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: []
W20260808 20:14:19.131470 14732 consensus_meta.cc:287] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.689s user 0.008s sys 0.000s
I20260808 20:14:19.135334 13077 raft_consensus.cc:1261] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 20:14:18.897094 14380 consensus_queue.cc:1046] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.351s
W20260808 20:14:19.124995 14743 log.cc:924] Time spent T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.360s user 0.000s sys 0.001s
I20260808 20:14:19.153653 12144 raft_consensus.cc:1226] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:19.155593 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:19.157230 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:19.123781 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 382251us
I20260808 20:14:18.919749 14738 consensus_queue.cc:1035] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.588s
W20260808 20:14:19.159245 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 631587us
I20260808 20:14:19.123602 14683 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:14:19.160286 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 49767.6us
I20260808 20:14:19.160929 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:19.161736 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:19.161669 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 78767.5us
I20260808 20:14:18.949412 12244 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:14:19.135392 13079 raft_consensus.cc:2454] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:19.180593 14397 consensus_queue.cc:1046] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 20:14:19.180840 14686 consensus_queue.cc:1035] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 20:14:19.186611 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:19.187290 12144 raft_consensus.cc:2362] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4. Re-sending same reply.
I20260808 20:14:19.187844 13077 raft_consensus.cc:1226] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:19.186734 14444 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:19.012248 14400 consensus_queue.cc:237] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:19.188949 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:19.159133 13080 rpcz_store.cc:276] 0808 20:14:17.958686 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:17.962030 (+ 3344us) service_pool.cc:227] Handling call
0808 20:14:17.977976 (+ 15946us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:17.978198 (+ 222us) raft_consensus.cc:1520] Early marking committed up to index 5
0808 20:14:18.140325 (+162127us) log.cc:842] Serialized 14 byte log entry
0808 20:14:19.116828 (+976503us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:19.116872 (+ 44us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:19.116932 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:19.116994 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:19.117089 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13,"prepare.queue_time_us":209087,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":4271,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":4292,"thread_start_us":1105204,"threads_started":3,"wal-append.queue_time_us":230130}
I20260808 20:14:19.189195 12105 leader_election.cc:302] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:19.167196 14575 raft_consensus.cc:2726] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:19.183413 12244 raft_consensus.cc:595] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:19.190610 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 232799us
I20260808 20:14:19.192473 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:19.167143 14732 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:19.206275 12152 raft_consensus.cc:2354] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for earlier term 2. Current term is 3.
I20260808 20:14:19.207119 14732 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:19.188282 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:19.162894 12036 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:19.139376 12254 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:19.195914 12147 raft_consensus.cc:2454] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:19.187738 14756 raft_consensus.cc:2781] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:19.199371 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:19.221766 12142 raft_consensus.cc:3037] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:19.160035 14680 consensus_queue.cc:1035] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20260808 20:14:19.190582 14686 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:19.202042 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
W20260808 20:14:19.205497 14744 log.cc:924] Time spent T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.388s user 0.000s sys 0.003s
I20260808 20:14:19.238202 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:19.246250 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:19.212409 12250 raft_consensus.cc:2454] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
W20260808 20:14:19.215246 13016 leader_election.cc:334] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.652s (SENT)
W20260808 20:14:19.217661 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 95082.5us
W20260808 20:14:19.260685 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 50207.3us
I20260808 20:14:19.261839 11994 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:19.222563 12205 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:19.223881 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:19.249013 13015 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:19.260114 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:19.263706 12147 raft_consensus.cc:1261] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 20:14:19.158753 14766 raft_consensus.cc:2781] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:19.265411 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:19.264144 12101 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:19.268538 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:19.221611 14756 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:14:19.269609 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 47163.7us
I20260808 20:14:19.278770 12205 leader_election.cc:302] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:19.270002 12101 leader_election.cc:302] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:19.280121 12101 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:19.214869 14777 raft_consensus.cc:2781] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:14:19.273980 12144 raft_consensus.cc:2379] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:19.274204 12244 raft_consensus.cc:1226] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:19.274438 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:19.274837 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:19.263880 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:19.224829 14533 raft_consensus.cc:2781] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:19.277537 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:19.293962 12152 raft_consensus.cc:1261] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:14:19.286612 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:19.291188 12147 raft_consensus.cc:1226] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:19.295650 11994 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:19.295918 12147 raft_consensus.cc:1226] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:19.297184 12147 raft_consensus.cc:1226] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:19.298920 11994 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:19.268489 14740 raft_consensus.cc:2781] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:19.312613 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:19.268807 14684 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:19.311159 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 43566us
I20260808 20:14:19.277009 14444 raft_consensus.cc:2726] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:19.291484 14732 raft_consensus.cc:2726] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260808 20:14:19.359932 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 82003.3us
W20260808 20:14:19.361477 13016 leader_election.cc:334] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.518s (SENT)
I20260808 20:14:19.361470 12101 leader_election.cc:302] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:19.363190 12245 raft_consensus.cc:1226] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:19.286361 14706 raft_consensus.cc:2726] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260808 20:14:19.333355 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 54650.7us
I20260808 20:14:19.384856 12205 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:19.385701 12205 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:19.276851 14734 consensus_queue.cc:1046] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20260808 20:14:19.285282 14756 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:19.387241 13016 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:19.263732 14686 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:19.276954 14669 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:19.393563 12250 raft_consensus.cc:1226] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:19.295244 14704 raft_consensus.cc:2726] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:19.301826 14777 raft_consensus.cc:686] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:19.393767 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:19.382712 14446 raft_consensus.cc:2726] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:19.382864 14740 raft_consensus.cc:686] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:19.403308 12245 raft_consensus.cc:1261] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 20:14:19.404709 12152 raft_consensus.cc:1226] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:19.387277 14684 raft_consensus.cc:2781] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:19.409240 12146 raft_consensus.cc:2379] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:19.382339 14782 raft_consensus.cc:2726] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:19.410905 12248 raft_consensus.cc:2379] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 5: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:19.393828 12138 raft_consensus.cc:1226] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: []
W20260808 20:14:19.413458 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 52007.9us
I20260808 20:14:19.394304 14706 raft_consensus.cc:2726] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260808 20:14:19.417598 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 56213.1us
I20260808 20:14:19.392189 12137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 220) took 1149 ms. Trace:
I20260808 20:14:19.414760 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:19.423847 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:19.424000 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:19.427069 14734 raft_consensus.cc:3032] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:19.427436 14734 raft_consensus.cc:729] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:19.428205 14734 consensus_queue.cc:260] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:19.429454 14734 raft_consensus.cc:3037] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
W20260808 20:14:19.431555 14755 log.cc:924] Time spent T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.161s user 0.000s sys 0.000s
I20260808 20:14:19.434005 14686 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:19.434077 14446 raft_consensus.cc:2781] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:19.419754 12137 rpcz_store.cc:276] 0808 20:14:18.238348 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:18.241572 (+ 3224us) service_pool.cc:227] Handling call
0808 20:14:19.333722 (+1092150us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b540030cd54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:19.382628 (+ 48906us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":401076992}
W20260808 20:14:19.435501 14753 log.cc:924] Time spent T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.254s user 0.000s sys 0.000s
I20260808 20:14:19.436880 14706 consensus_queue.cc:1046] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:14:19.437269 14704 consensus_queue.cc:1046] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:19.438177 14706 raft_consensus.cc:3032] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:19.434827 14446 raft_consensus.cc:686] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:19.438326 14704 raft_consensus.cc:3032] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:19.438982 14704 raft_consensus.cc:729] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:19.404696 14533 raft_consensus.cc:686] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:19.439764 14704 consensus_queue.cc:260] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:19.440831 14704 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:19.441308 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:19.416716 14673 raft_consensus.cc:2726] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:19.439460 14446 consensus_queue.cc:237] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:19.447965 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:19.448928 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:19.448493 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:19.450340 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:19.452023 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:19.452732 12040 raft_consensus.cc:2379] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:19.440119 14533 consensus_queue.cc:237] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:19.440419 11889 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:57474 (request call id 332) took 10889 ms. Trace:
I20260808 20:14:19.410192 12241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1603) took 1002 ms. Trace:
I20260808 20:14:19.448550 12039 raft_consensus.cc:2454] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:19.469499 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:19.474366 12040 raft_consensus.cc:2354] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4.
I20260808 20:14:19.467696 13016 leader_election.cc:302] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:19.409655 14786 raft_consensus.cc:2726] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:19.475121 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:19.434741 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:19.475224 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:19.416013 14684 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:19.489581 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:19.490213 12039 raft_consensus.cc:2362] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3. Re-sending same reply.
I20260808 20:14:19.408483 14741 consensus_queue.cc:1046] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.064s
I20260808 20:14:19.446988 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:19.438519 14706 raft_consensus.cc:729] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
W20260808 20:14:19.477583 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 91942.7us
I20260808 20:14:19.471299 11889 rpcz_store.cc:276] 0808 20:14:08.551425 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:08.551553 (+ 128us) service_pool.cc:227] Handling call
0808 20:14:19.440360 (+10888807us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 20:14:18.171429 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 20:14:18.184875 (+ 13446us) write_op.cc:432] Acquiring schema lock in shared mode
0808 20:14:18.188965 (+ 4090us) write_op.cc:435] Acquired schema lock
0808 20:14:18.188983 (+ 18us) tablet.cc:660] Decoding operations
0808 20:14:18.201882 (+ 12899us) write_op.cc:620] Acquiring the partition lock for write op
0808 20:14:18.201983 (+ 101us) write_op.cc:641] Partition lock acquired for write op
0808 20:14:18.202010 (+ 27us) tablet.cc:683] Acquiring locks for 3 operations
0808 20:14:18.202452 (+ 442us) tablet.cc:699] Row locks acquired
0808 20:14:18.202502 (+ 50us) write_op.cc:260] PREPARE: finished
0808 20:14:18.202677 (+ 175us) write_op.cc:270] Start()
0808 20:14:18.202825 (+ 148us) write_op.cc:276] Timestamp: P: 1786220058202619 usec, L: 0
0808 20:14:18.202862 (+ 37us) op_driver.cc:331] REPLICATION: starting
0808 20:14:18.203467 (+ 605us) log.cc:842] Serialized 1289 byte log entry
0808 20:14:18.383336 (+179869us) op_driver.cc:444] REPLICATION: finished
0808 20:14:18.471306 (+ 87970us) write_op.cc:301] APPLY: starting
0808 20:14:18.471400 (+ 94us) tablet.cc:1365] starting BulkCheckPresence
0808 20:14:18.471524 (+ 124us) tablet.cc:1368] finished BulkCheckPresence
0808 20:14:18.479212 (+ 7688us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 20:14:18.483587 (+ 4375us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 20:14:18.483803 (+ 216us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0808 20:14:18.483846 (+ 43us) write_op.cc:312] APPLY: finished
0808 20:14:18.484127 (+ 281us) log.cc:842] Serialized 35 byte log entry
0808 20:14:18.484282 (+ 155us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:14:18.484369 (+ 87us) write_op.cc:454] Released schema lock
0808 20:14:18.484408 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0808 20:14:18.484670 (+ 262us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:14:18.484695 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":39150,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":87694,"apply.run_cpu_time_us":2802,"apply.run_wall_time_us":13655,"num_ops":3,"prepare.queue_time_us":285765,"prepare.run_cpu_time_us":3858,"prepare.run_wall_time_us":47972,"replication_time_us":180381,"thread_start_us":242352,"threads_started":2}]]}
I20260808 20:14:19.479329 12000 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:19.473477 12241 rpcz_store.cc:276] 0808 20:14:18.408199 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:18.410195 (+ 1996us) service_pool.cc:227] Handling call
0808 20:14:19.393958 (+983763us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b5400087854. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:19.406427 (+ 12469us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39858432}
I20260808 20:14:19.480012 12040 raft_consensus.cc:2379] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:19.481802 14669 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:19.418077 14787 raft_consensus.cc:2781] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:19.475276 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:19.502645 13016 leader_election.cc:302] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:19.491503 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:19.492110 12249 raft_consensus.cc:2454] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:19.494263 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:19.499433 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:19.504338 12040 raft_consensus.cc:2362] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3. Re-sending same reply.
I20260808 20:14:19.504971 12038 raft_consensus.cc:1226] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:19.475895 14686 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:19.500377 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:19.509045 12040 raft_consensus.cc:1261] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260808 20:14:19.507782 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 208904us
I20260808 20:14:19.500500 12205 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:19.510794 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 211517us
I20260808 20:14:19.512434 12043 raft_consensus.cc:1261] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 20:14:19.512951 12038 raft_consensus.cc:1226] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:19.512928 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:19.514297 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 119719us
I20260808 20:14:19.510655 12000 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:19.488245 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:19.517370 12039 raft_consensus.cc:2379] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 3: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:19.493290 14684 raft_consensus.cc:3037] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:19.522774 13016 leader_election.cc:302] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:19.522820 12000 leader_election.cc:302] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:19.513535 11994 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:19.528939 12000 leader_election.cc:302] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:19.501976 14673 raft_consensus.cc:3037] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:19.533898 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:19.532231 12000 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:19.501096 14787 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:19.533087 12048 raft_consensus.cc:2379] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:19.499910 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:19.513441 14786 raft_consensus.cc:2726] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260808 20:14:19.578208 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 64767.8us
I20260808 20:14:19.558480 14575 raft_consensus.cc:2726] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:19.579896 12036 raft_consensus.cc:2379] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 3: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:19.529740 14680 raft_consensus.cc:2726] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:19.525699 14741 raft_consensus.cc:2726] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:19.525972 14799 raft_consensus.cc:2726] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:19.532652 14782 raft_consensus.cc:2726] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:19.590708 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:19.593205 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 70508.6us
I20260808 20:14:19.519086 14686 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:19.513113 14734 consensus_queue.cc:1046] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20260808 20:14:19.578799 14787 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:19.597453 13016 leader_election.cc:302] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:19.593297 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 63939.7us
I20260808 20:14:19.584753 14468 raft_consensus.cc:3037] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:19.585860 14766 raft_consensus.cc:686] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:19.597380 14793 raft_consensus.cc:2726] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:19.520605 14740 consensus_queue.cc:237] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:19.583580 12141 raft_consensus.cc:2454] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:14:19.611982 12141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1218) took 1078 ms. Trace:
I20260808 20:14:19.612098 14686 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:19.596958 14575 raft_consensus.cc:2726] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:19.613773 12142 raft_consensus.cc:2454] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:19.603791 14680 raft_consensus.cc:482] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:19.614470 14680 raft_consensus.cc:504] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:19.616343 14680 leader_election.cc:288] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:19.609056 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 76898.8us
I20260808 20:14:19.617923 12000 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:19.624158 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:19.624804 13077 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:19.622491 12000 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:19.638516 12001 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:19.640309 12000 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:19.520603 14777 consensus_queue.cc:237] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:19.642326 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 110277us
I20260808 20:14:19.642339 12249 raft_consensus.cc:1261] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 20:14:19.642511 14793 raft_consensus.cc:2781] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:19.648113 12205 leader_election.cc:302] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:19.649183 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 51804.3us
I20260808 20:14:19.616994 14674 raft_consensus.cc:2781] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:19.651199 13016 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:19.648105 12102 leader_election.cc:302] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:19.637955 14786 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:19.589059 14706 consensus_queue.cc:260] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:19.639972 14575 raft_consensus.cc:2781] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:19.638761 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:19.647352 14686 raft_consensus.cc:2781] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:19.667114 12000 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:19.605953 14446 consensus_queue.cc:1046] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20260808 20:14:19.673184 12102 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:19.674036 14801 consensus_queue.cc:1035] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260808 20:14:19.675559 14801 raft_consensus.cc:2726] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:19.660090 14786 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:19.677417 14786 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:19.653743 14674 raft_consensus.cc:482] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:19.668869 14575 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:19.665978 14704 raft_consensus.cc:2726] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:19.617369 12141 rpcz_store.cc:276] 0808 20:14:18.533354 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:18.535304 (+ 1950us) service_pool.cc:227] Handling call
0808 20:14:19.497228 (+961924us) spinlock_profiling.cc:243] Waited 1.85 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537ab7c 00007fa96537a73b 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:19.611921 (+114693us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4266368}
I20260808 20:14:19.684578 13078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 353) took 1141 ms. Trace:
I20260808 20:14:19.686136 13078 rpcz_store.cc:276] 0808 20:14:18.542710 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:18.542883 (+ 173us) service_pool.cc:227] Handling call
0808 20:14:18.543134 (+ 251us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:18.543298 (+ 164us) raft_consensus.cc:1520] Early marking committed up to index 5
0808 20:14:18.591233 (+ 47935us) log.cc:842] Serialized 14 byte log entry
0808 20:14:19.684183 (+1092950us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x7b18003d5d00. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96e2518b1 00007fa96e25e661 00007fa967f6982c 00007fa9653949dc 00007fa96e1cba39 00007fa9653b18bb 00007fa9653afcbf 00007fa9653af155 00007fa9682d23d0 00007fa9682e4576 00007fa9682f50d7
0808 20:14:19.684274 (+ 91us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:19.684320 (+ 46us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:19.684361 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:19.684418 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:19.684524 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":46797,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":93082,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":9693,"spinlock_wait_cycles":31207808,"thread_start_us":200649,"threads_started":1}
I20260808 20:14:19.687249 14793 raft_consensus.cc:686] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:19.670182 14686 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:19.688095 14686 raft_consensus.cc:3037] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:19.688287 14793 consensus_queue.cc:237] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:19.681713 14674 raft_consensus.cc:3037] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:19.698645 14758 raft_consensus.cc:3032] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:19.699020 14758 raft_consensus.cc:729] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:19.697487 14810 raft_consensus.cc:3032] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:19.683349 14706 raft_consensus.cc:3037] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:19.685979 14575 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:19.684981 14794 raft_consensus.cc:2726] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:19.704496 12102 leader_election.cc:302] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:19.693678 14680 raft_consensus.cc:482] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
W20260808 20:14:19.669917 14732 consensus_queue.cc:1159] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" responder_term: 3 status { last_received { term: 3 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 3 index: 3 } }
I20260808 20:14:19.699793 14758 consensus_queue.cc:260] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:19.693761 14786 raft_consensus.cc:2781] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:19.710265 11894 catalog_manager.cc:5495] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 1 to 2, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 2 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:19.712203 14397 consensus_queue.cc:1046] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 20:14:19.693849 12243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1618) took 1074 ms. Trace:
I20260808 20:14:19.702976 14810 raft_consensus.cc:729] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:19.718473 14810 consensus_queue.cc:260] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:19.719762 14810 raft_consensus.cc:3037] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:19.715054 14794 raft_consensus.cc:979] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a: : Instructing follower 78b9263889cd4f1582578a3aa4f644d9 to start an election
I20260808 20:14:19.716646 11894 catalog_manager.cc:5495] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 2 to 3, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 3 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:14:19.714898 14704 raft_consensus.cc:2726] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:19.724581 12102 leader_election.cc:302] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:19.729063 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 69173.3us
I20260808 20:14:19.733314 14824 consensus_queue.cc:1035] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
W20260808 20:14:19.734629 14683 consensus_meta.cc:287] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.560s user 0.007s sys 0.003s
I20260808 20:14:19.735034 14683 raft_consensus.cc:2726] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901) with higher term. Message: Invalid argument: T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4.
I20260808 20:14:19.736043 12140 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:19.666703 14766 consensus_queue.cc:237] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:19.715654 14723 log.cc:924] Time spent T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.846s user 0.000s sys 0.012s
I20260808 20:14:19.743521 14809 raft_consensus.cc:2726] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:19.749490 14733 raft_consensus.cc:1067] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Signalling peer 78b9263889cd4f1582578a3aa4f644d9 to start an election
I20260808 20:14:19.751377 14444 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:19.756932 14788 raft_consensus.cc:2726] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:19.757002 14444 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:19.759112 14444 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:19.760886 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:19.761785 14684 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:19.709906 14793 consensus_queue.cc:1046] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260808 20:14:19.725574 14786 raft_consensus.cc:686] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:19.813479 14809 raft_consensus.cc:3032] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:19.813891 14809 raft_consensus.cc:729] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:19.813654 14786 consensus_queue.cc:237] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:19.814679 14809 consensus_queue.cc:260] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:19.801028 14795 raft_consensus.cc:2726] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 20:14:19.820554 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 71259us
I20260808 20:14:19.801757 14673 raft_consensus.cc:2726] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:19.802780 14684 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:19.817679 12102 leader_election.cc:398] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747) with higher term. Message: Invalid argument: T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 4. Current term is 5.
W20260808 20:14:19.822331 11994 leader_election.cc:334] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Timed out: RequestConsensusVote RPC to 127.10.24.3:38391 timed out after 1.608s (SENT)
I20260808 20:14:19.822901 12102 leader_election.cc:401] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260808 20:14:19.812264 14674 raft_consensus.cc:504] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:19.715672 12243 rpcz_store.cc:276] 0808 20:14:18.611209 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:18.625190 (+ 13981us) service_pool.cc:227] Handling call
0808 20:14:19.616869 (+991679us) spinlock_profiling.cc:243] Waited 8.48 ms on lock 0x7b58001a1398. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:19.683820 (+ 66951us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19498368}
W20260808 20:14:19.883301 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 61229.7us
I20260808 20:14:19.737076 14680 raft_consensus.cc:504] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:19.885200 12205 leader_election.cc:302] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:19.886238 13078 raft_consensus.cc:1226] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 2. Current term is 3. Ops: []
W20260808 20:14:19.887514 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 204988us
I20260808 20:14:19.885855 14355 raft_consensus.cc:482] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:19.891047 14680 leader_election.cc:288] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:19.891649 14789 raft_consensus.cc:482] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:19.892441 14674 leader_election.cc:288] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:19.892138 14789 raft_consensus.cc:504] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:19.894119 14789 leader_election.cc:288] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:19.808400 14787 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:19.899075 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:19.884670 14795 raft_consensus.cc:2726] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747) with higher term. Message: Invalid argument: T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 4. Current term is 5.
I20260808 20:14:19.907897 12038 raft_consensus.cc:1261] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 20:14:19.910113 14575 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:19.913038 14817 consensus_queue.cc:1046] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260808 20:14:19.914094 14817 raft_consensus.cc:3032] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:19.914284 14758 raft_consensus.cc:3037] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:19.889355 14355 raft_consensus.cc:504] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:19.914608 14686 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:19.916100 14355 leader_election.cc:288] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:19.916388 14686 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:19.917218 14575 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:19.921638 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 172938us
I20260808 20:14:19.959764 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:19.900584 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:19.905640 14810 raft_consensus.cc:2726] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 20:14:19.968169 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 317035us
I20260808 20:14:19.906985 12243 raft_consensus.cc:1261] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 20:14:19.971844 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:19.972882 12043 raft_consensus.cc:2379] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 4: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:19.897435 14824 consensus_queue.cc:1046] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
W20260808 20:14:19.975344 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 52998us
I20260808 20:14:19.921597 14817 raft_consensus.cc:729] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:19.981985 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 335) took 1488 ms. Trace:
I20260808 20:14:19.982407 12041 rpcz_store.cc:276] 0808 20:14:18.493296 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:18.497505 (+ 4209us) service_pool.cc:227] Handling call
0808 20:14:19.149167 (+651662us) spinlock_profiling.cc:243] Waited 4.15 ms on lock 0x7b5800001898. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:19.149358 (+ 191us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:19.172752 (+ 23394us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:19.181960 (+ 9208us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:19.184034 (+ 2074us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:19.190420 (+ 6386us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:19.981742 (+791322us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b540012c5d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:19.981784 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:19.981898 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40983168}
I20260808 20:14:19.987793 14630 raft_consensus.cc:3032] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:19.988173 14630 raft_consensus.cc:729] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:19.988910 14630 consensus_queue.cc:260] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:19.989832 14630 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:19.975582 14355 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
I20260808 20:14:19.967650 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:19.997931 14787 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:20.001189 12102 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:19.894461 14756 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:20.013924 14756 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:20.014503 12251 raft_consensus.cc:1226] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5. Ops: []
W20260808 20:14:20.019548 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 43592.9us
I20260808 20:14:20.020720 12102 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.021090 14704 raft_consensus.cc:2726] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:20.021420 14795 raft_consensus.cc:2726] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:20.005105 14468 raft_consensus.cc:2726] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:20.022004 14795 consensus_queue.cc:1035] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:14:20.006594 12140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1508) took 1094 ms. Trace:
I20260808 20:14:20.022660 12140 rpcz_store.cc:276] 0808 20:14:18.912085 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:19.188418 (+276333us) service_pool.cc:227] Handling call
0808 20:14:20.006416 (+817998us) spinlock_profiling.cc:243] Waited 537 ms on lock 0x7b5400106654. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:20.006529 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1235710720}
I20260808 20:14:20.023551 13075 raft_consensus.cc:1226] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:20.023874 12102 leader_election.cc:302] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.024672 14795 raft_consensus.cc:2726] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:20.025017 13075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 349) took 1483 ms. Trace:
I20260808 20:14:20.025884 12102 leader_election.cc:302] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.006978 12146 raft_consensus.cc:1261] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 20:14:20.007692 12138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1503) took 1509 ms. Trace:
I20260808 20:14:20.037829 12138 rpcz_store.cc:276] 0808 20:14:18.497665 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:18.500472 (+ 2807us) service_pool.cc:227] Handling call
0808 20:14:19.382646 (+882174us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:19.449537 (+ 66891us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:20.007334 (+557797us) spinlock_profiling.cc:243] Waited 776 ms on lock 0x7b540030cd50. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d9a09 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:20.007479 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1783959808}
I20260808 20:14:20.006625 12041 raft_consensus.cc:1226] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:20.039357 14795 raft_consensus.cc:2726] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:20.009946 14676 raft_consensus.cc:3032] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:20.025352 13075 rpcz_store.cc:276] 0808 20:14:18.541449 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:18.590908 (+ 49459us) service_pool.cc:227] Handling call
0808 20:14:19.674023 (+1083115us) spinlock_profiling.cc:243] Waited 21.1 ms on lock 0x7b5800005198. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:19.674231 (+ 208us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:20.023914 (+349683us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:20.024784 (+ 870us) spinlock_profiling.cc:243] Waited 349 ms on lock 0x7b540024fed4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:20.024961 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":851465216}
I20260808 20:14:20.040100 14676 raft_consensus.cc:729] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:20.011831 14817 consensus_queue.cc:260] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:20.041149 14676 consensus_queue.cc:260] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:20.042421 14817 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:20.042793 14676 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:20.044867 12244 raft_consensus.cc:1226] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:20.040294 12047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1215) took 1442 ms. Trace:
I20260808 20:14:20.046607 12047 rpcz_store.cc:276] 0808 20:14:18.598316 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:18.909383 (+311067us) service_pool.cc:227] Handling call
0808 20:14:18.928363 (+ 18980us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:18.928569 (+ 206us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:18.928596 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:18.929004 (+ 408us) log.cc:842] Serialized 26 byte log entry
0808 20:14:19.192838 (+263834us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:19.262113 (+ 69275us) log.cc:842] Serialized 14 byte log entry
0808 20:14:19.268117 (+ 6004us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:19.268185 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:20.015512 (+747327us) spinlock_profiling.cc:243] Waited 45.2 ms on lock 0x7b1800284920. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96e2518b1 00007fa96e25e661 00007fa9653b2638 00007fa9653b5b11 00007fa9653b5ac9 00007fa9653b5a59 00007fa9653b5a21 00007fa9653b4d1d 00007fa96e616204 00007fa96e616039 00007fa965394dc6
0808 20:14:20.040033 (+ 24521us) raft_consensus.cc:1671] finished
0808 20:14:20.040128 (+ 95us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:20.040242 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2911,"prepare.queue_time_us":31456,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":29577,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":23149,"spinlock_wait_cycles":103915264,"thread_start_us":314635,"threads_started":2,"wal-append.queue_time_us":380111}
I20260808 20:14:20.048636 14351 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:20.049239 14351 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:20.051482 14351 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:20.052091 14351 consensus_queue.cc:1046] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:20.052348 12251 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1626) took 1287 ms. Trace:
I20260808 20:14:20.053278 14501 raft_consensus.cc:3032] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:20.053692 14501 raft_consensus.cc:729] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:20.053958 14807 raft_consensus.cc:3032] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:20.054325 14807 raft_consensus.cc:729] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:20.054530 14501 consensus_queue.cc:260] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:20.055660 14501 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:20.055456 14807 consensus_queue.cc:260] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:20.062556 14684 raft_consensus.cc:3032] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:20.062924 14684 raft_consensus.cc:729] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:20.063683 14684 consensus_queue.cc:260] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:20.036660 12102 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.052716 12251 rpcz_store.cc:276] 0808 20:14:18.764491 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:18.764629 (+ 138us) service_pool.cc:227] Handling call
0808 20:14:19.206598 (+441969us) spinlock_profiling.cc:243] Waited 6.26 ms on lock 0x7b58001a1398. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:19.206794 (+ 196us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:20.014881 (+808087us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:20.052119 (+ 37238us) spinlock_profiling.cc:243] Waited 807 ms on lock 0x7b5400143f54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:20.052270 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1871584384}
W20260808 20:14:20.068759 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 95309.4us
I20260808 20:14:20.069698 13016 leader_election.cc:302] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.073118 12102 leader_election.cc:302] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.075032 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:20.077005 14809 raft_consensus.cc:3037] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:20.042940 14800 raft_consensus.cc:3032] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:20.082495 13079 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:20.084537 12250 raft_consensus.cc:1261] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260808 20:14:20.084918 12250 pending_rounds.cc:76] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a: Aborting all ops after (but not including) 4
I20260808 20:14:20.085139 12250 pending_rounds.cc:97] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260808 20:14:20.085363 12250 raft_consensus.cc:2866] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 20:14:20.085634 12248 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1619) took 1474 ms (client timeout 1751 ms). Trace:
W20260808 20:14:20.086014 12248 rpcz_store.cc:269] 0808 20:14:18.611506 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:18.611638 (+ 132us) service_pool.cc:227] Handling call
0808 20:14:20.085479 (+1473841us) spinlock_profiling.cc:243] Waited 798 ms on lock 0x7b54000bb8d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:20.085587 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1836026624}
I20260808 20:14:20.089818 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:20.090469 12043 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:20.098409 12039 raft_consensus.cc:1226] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:20.090026 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:20.099212 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 393461us
I20260808 20:14:20.081475 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:20.064790 14684 raft_consensus.cc:3037] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:20.104444 12206 leader_election.cc:302] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:20.105547 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:20.106559 14351 raft_consensus.cc:2726] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:20.107189 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:20.107627 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:20.108217 12241 raft_consensus.cc:2454] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:20.108563 12041 raft_consensus.cc:1226] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260808 20:14:20.051397 14683 consensus_queue.cc:1035] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
W20260808 20:14:20.117801 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 44722.8us
I20260808 20:14:20.119099 12102 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.121104 12205 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:20.084489 14800 raft_consensus.cc:729] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:20.123008 12001 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:20.100389 14706 raft_consensus.cc:2726] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 20:14:20.126252 12205 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:20.126613 12206 leader_election.cc:302] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:20.104585 14795 raft_consensus.cc:2726] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:20.135016 12102 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:20.139103 12102 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.150925 13014 leader_election.cc:302] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:20.117147 14799 raft_consensus.cc:2726] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:20.153412 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:20.158483 13014 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.156728 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:20.125818 14683 raft_consensus.cc:2726] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 20:14:20.164777 12102 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.114534 14817 consensus_queue.cc:1035] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260808 20:14:20.133632 14793 raft_consensus.cc:2781] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:20.135741 14738 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:20.136356 14432 raft_consensus.cc:2726] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:14:20.168221 12105 leader_election.cc:302] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:20.147853 14533 raft_consensus.cc:3037] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:20.136037 14760 raft_consensus.cc:482] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:20.147643 14704 raft_consensus.cc:2726] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:20.196750 12150 raft_consensus.cc:1226] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 LEADER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:20.124155 14446 consensus_queue.cc:1046] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
W20260808 20:14:20.192317 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 68810.9us
I20260808 20:14:20.160755 14822 raft_consensus.cc:2726] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:20.198596 12248 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:14:20.202257 12248 raft_consensus.cc:595] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:20.201643 12206 leader_election.cc:398] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Vote denied by peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139) with higher term. Message: Invalid argument: T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for earlier term 5. Current term is 6.
I20260808 20:14:20.189139 14355 raft_consensus.cc:2726] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:20.191342 14793 raft_consensus.cc:686] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:20.197327 14683 raft_consensus.cc:2726] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:20.210817 14840 raft_consensus.cc:2781] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260808 20:14:20.211463 14827 log.cc:924] Time spent T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.237s user 0.000s sys 0.000s
I20260808 20:14:20.219916 14808 raft_consensus.cc:1050] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
W20260808 20:14:20.220304 14808 raft_consensus.cc:1054] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995: Unable to promote non-voter e5bd9a4616ce4acfa20dc7a002984a20: Illegal state: Leader has not yet committed an operation in its own term
I20260808 20:14:20.221375 14680 raft_consensus.cc:1050] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
W20260808 20:14:20.221750 14680 raft_consensus.cc:1054] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995: Unable to promote non-voter e5bd9a4616ce4acfa20dc7a002984a20: Illegal state: Leader has not yet committed an operation in its own term
I20260808 20:14:20.222258 14807 raft_consensus.cc:3037] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:20.222657 12150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1235) took 1214 ms. Trace:
I20260808 20:14:20.202481 14777 raft_consensus.cc:2726] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:20.239567 12046 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:14:20.223023 12150 rpcz_store.cc:276] 0808 20:14:19.008491 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:19.263011 (+254520us) service_pool.cc:227] Handling call
0808 20:14:19.263257 (+ 246us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:20.197174 (+933917us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:20.222449 (+ 25275us) spinlock_profiling.cc:243] Waited 933 ms on lock 0x7b5400061154. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:20.222601 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2146455424}
I20260808 20:14:20.240209 12046 raft_consensus.cc:595] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:20.165302 14733 consensus_queue.cc:1046] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260808 20:14:20.207857 12241 raft_consensus.cc:1226] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260808 20:14:20.226078 12043 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:20.200265 14760 raft_consensus.cc:504] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:20.255800 14760 leader_election.cc:288] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:20.257243 14704 raft_consensus.cc:1050] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:14:20.197520 14800 consensus_queue.cc:260] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:20.267072 14800 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:20.267635 14733 raft_consensus.cc:3032] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:20.268030 14733 raft_consensus.cc:729] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:20.269287 14733 consensus_queue.cc:260] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:20.238318 14793 consensus_queue.cc:237] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:20.240569 12046 raft_consensus.cc:603] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 20:14:20.273062 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:20.246287 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 44887us
I20260808 20:14:20.224538 14840 raft_consensus.cc:482] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:20.264943 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:20.274935 12206 leader_election.cc:302] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:20.279451 14683 consensus_queue.cc:1046] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 20:14:20.282435 12205 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:20.282666 13015 leader_election.cc:302] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.280755 14683 raft_consensus.cc:3032] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:20.288628 14836 raft_consensus.cc:2726] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:20.290210 14444 raft_consensus.cc:2781] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:20.290784 14444 raft_consensus.cc:686] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:20.292306 14444 consensus_queue.cc:237] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:20.295379 12245 raft_consensus.cc:1261] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:14:20.296370 14738 raft_consensus.cc:2726] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:20.297235 14734 raft_consensus.cc:2726] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:20.298244 14706 consensus_queue.cc:1035] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:20.298646 14396 consensus_queue.cc:1046] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:20.299646 12206 leader_election.cc:398] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Vote denied by peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139) with higher term. Message: Invalid argument: T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for earlier term 2. Current term is 3.
I20260808 20:14:20.334532 12206 leader_election.cc:401] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260808 20:14:20.287997 14683 raft_consensus.cc:729] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:20.274580 14840 raft_consensus.cc:3037] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:20.307782 14789 raft_consensus.cc:482] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:20.308158 14533 raft_consensus.cc:2726] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:20.287493 14733 raft_consensus.cc:3037] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:20.311685 14704 consensus_queue.cc:237] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:20.333666 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 168942us
I20260808 20:14:20.347272 12102 leader_election.cc:302] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.346122 13077 raft_consensus.cc:1226] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:20.353137 14801 raft_consensus.cc:2781] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:20.353629 14801 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:20.354051 14801 raft_consensus.cc:3037] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:20.358219 14738 raft_consensus.cc:3032] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:20.358651 14738 raft_consensus.cc:729] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:20.358740 12241 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:14:20.359421 12241 raft_consensus.cc:595] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:20.361433 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260808 20:14:20.361505 12102 leader_election.cc:398] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747) with higher term. Message: Invalid argument: T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4.
I20260808 20:14:20.363184 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:14:20.363986 12102 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.364830 14850 raft_consensus.cc:2726] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:20.365139 14734 raft_consensus.cc:2932] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:20.366345 12102 leader_election.cc:302] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:20.367728 14801 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:20.367523 12046 raft_consensus.cc:2932] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
W20260808 20:14:20.267194 14407 consensus_queue.cc:1159] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" responder_term: 4 status { last_received { term: 4 index: 4 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260808 20:14:20.371574 14850 raft_consensus.cc:2726] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:20.333743 14732 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:20.380304 13077 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEARNER]: Advancing to term 4
I20260808 20:14:20.378500 12102 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.401649 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:14:20.358119 14396 raft_consensus.cc:2726] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139) with higher term. Message: Invalid argument: T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for earlier term 2. Current term is 3.
I20260808 20:14:20.406256 12045 raft_consensus.cc:1226] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: []
W20260808 20:14:20.407191 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 107715us
I20260808 20:14:20.396174 13077 raft_consensus.cc:1261] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEARNER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 20:14:20.348922 14789 raft_consensus.cc:504] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:20.385300 14732 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:20.441723 14789 leader_election.cc:288] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:20.443259 14732 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:20.447556 12102 leader_election.cc:302] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.404109 14795 raft_consensus.cc:2726] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 20:14:20.436298 14817 consensus_queue.cc:1046] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20260808 20:14:20.406929 14683 consensus_queue.cc:260] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:20.450613 14683 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:20.437842 14396 consensus_queue.cc:1035] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20260808 20:14:20.439389 12046 raft_consensus.cc:1261] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 20:14:20.443665 12206 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:20.465211 14836 consensus_queue.cc:1035] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:20.359516 14738 consensus_queue.cc:260] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:20.468667 14801 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:20.463869 14826 raft_consensus.cc:3032] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:20.493392 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:20.496279 12102 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:20.498236 12102 leader_election.cc:302] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.502770 14732 raft_consensus.cc:2726] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:20.504686 14801 consensus_queue.cc:1035] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260808 20:14:20.505746 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 62183.7us
I20260808 20:14:20.506043 14801 raft_consensus.cc:2726] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:20.506577 14799 raft_consensus.cc:2781] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:20.506954 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:20.477763 14826 raft_consensus.cc:729] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:20.509219 14787 consensus_queue.cc:1046] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:20.509513 14826 consensus_queue.cc:260] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:20.510262 14787 raft_consensus.cc:3032] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:20.510605 14787 raft_consensus.cc:729] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:20.510990 14738 raft_consensus.cc:3037] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:20.511376 14787 consensus_queue.cc:260] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:20.478281 14777 raft_consensus.cc:2726] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260808 20:14:20.513098 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 78269.6us
I20260808 20:14:20.518579 12150 raft_consensus.cc:1261] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 20:14:20.518685 11994 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:20.525024 11994 leader_election.cc:302] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:20.532590 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:20.533170 12142 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:20.525995 12150 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:53162
I20260808 20:14:20.533895 12150 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:14:20.534220 12150 raft_consensus.cc:3037] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:20.512733 14787 raft_consensus.cc:3037] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:20.512501 14840 raft_consensus.cc:504] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:20.506956 14799 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:20.514032 14706 consensus_queue.cc:1046] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260808 20:14:20.544026 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:20.544740 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:20.543288 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:20.520512 12243 raft_consensus.cc:2454] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:20.523083 14793 raft_consensus.cc:2781] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:20.532428 14809 raft_consensus.cc:2781] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:20.535126 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:20.584511 12146 raft_consensus.cc:2379] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 5: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:20.589023 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:20.590197 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:20.591218 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
W20260808 20:14:20.584748 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 45676.2us
I20260808 20:14:20.591948 12241 raft_consensus.cc:2379] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 5: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:20.593307 14706 raft_consensus.cc:3032] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:20.593547 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:20.594285 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:20.594810 12245 raft_consensus.cc:1226] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:20.595718 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:20.596597 12141 raft_consensus.cc:2396] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 20:14:20.596700 14683 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:20.520606 14800 consensus_queue.cc:1035] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.062s
I20260808 20:14:20.593724 14706 raft_consensus.cc:729] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:20.592681 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:20.621237 12150 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:20.597278 14683 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:20.606683 12142 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
W20260808 20:14:20.623220 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 238927us
I20260808 20:14:20.542405 14799 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:20.657584 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:20.664953 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 166777us
I20260808 20:14:20.674113 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:20.685815 13077 raft_consensus.cc:2379] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 3: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:20.685853 12102 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.620445 14793 raft_consensus.cc:686] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:20.599666 14809 raft_consensus.cc:686] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:20.588116 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:20.693092 12150 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:20.695223 12251 raft_consensus.cc:1261] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 20:14:20.700696 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:20.701308 11994 leader_election.cc:302] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:20.704679 11994 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.708793 14680 raft_consensus.cc:2726] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:20.709450 12141 raft_consensus.cc:1226] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:20.710860 12141 raft_consensus.cc:1261] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 20:14:20.713341 14732 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:20.713786 14732 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:20.717129 14786 raft_consensus.cc:2726] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:20.717486 12254 raft_consensus.cc:1261] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 20:14:20.722254 14732 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:20.723142 12250 raft_consensus.cc:2454] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:20.726612 12251 raft_consensus.cc:2454] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:20.726828 14683 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:20.727492 12241 raft_consensus.cc:2454] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:20.731281 12206 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:20.736567 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 70109.8us
I20260808 20:14:20.738190 12102 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.743395 12102 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:20.744084 13077 raft_consensus.cc:1261] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 LEARNER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:14:20.744570 12206 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:20.688678 14831 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:20.599738 14818 consensus_queue.cc:1046] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.088s
I20260808 20:14:20.701682 14850 raft_consensus.cc:2726] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 20:14:20.747756 12152 raft_consensus.cc:1226] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:20.772835 12251 raft_consensus.cc:1261] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 20:14:20.774530 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:20.747110 14400 raft_consensus.cc:2726] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:20.784180 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:20.786505 12150 raft_consensus.cc:2379] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 2: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:20.787289 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:20.770202 14777 raft_consensus.cc:2726] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:20.770030 14533 raft_consensus.cc:2781] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:20.789880 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:20.740499 14733 consensus_queue.cc:1035] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260808 20:14:20.791460 12150 raft_consensus.cc:2454] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:20.772266 14738 raft_consensus.cc:2726] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:14:20.788836 12046 raft_consensus.cc:1226] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260808 20:14:20.740842 14786 consensus_queue.cc:1035] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260808 20:14:20.795002 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:20.794025 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:20.780332 13073 raft_consensus.cc:1261] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:14:20.796264 12150 raft_consensus.cc:3037] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:20.732707 14706 consensus_queue.cc:260] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:20.777050 14795 consensus_queue.cc:1035] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20260808 20:14:20.777900 14432 consensus_queue.cc:1046] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20260808 20:14:20.797646 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:20.783943 14756 consensus_queue.cc:1035] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260808 20:14:20.771713 14831 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:20.793859 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
W20260808 20:14:20.811944 13016 leader_election.cc:334] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.528s (SENT)
I20260808 20:14:20.815510 13016 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:20.819712 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:20.820310 12250 raft_consensus.cc:3037] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:20.799844 14788 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:20.803947 14533 raft_consensus.cc:686] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 5 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:20.782335 14793 consensus_queue.cc:237] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:20.823374 14864 log.cc:924] Time spent T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.120s user 0.001s sys 0.000s
I20260808 20:14:20.821429 14840 leader_election.cc:288] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:20.824129 14706 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:20.835422 12150 raft_consensus.cc:2454] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:20.831236 12250 raft_consensus.cc:2454] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:20.834081 12141 raft_consensus.cc:2379] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
W20260808 20:14:20.848798 13016 leader_election.cc:334] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.587s (SENT)
I20260808 20:14:20.851884 13014 leader_election.cc:302] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:20.857666 13016 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.807817 14777 consensus_queue.cc:1035] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260808 20:14:20.796240 14809 consensus_queue.cc:237] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:20.828284 14468 raft_consensus.cc:2726] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:20.859313 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:20.900867 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:20.901212 12150 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:20.901736 14396 raft_consensus.cc:3032] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:20.902070 14396 raft_consensus.cc:729] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:20.902885 14396 consensus_queue.cc:260] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:20.904644 12137 raft_consensus.cc:1226] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:20.905704 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:20.906836 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:20.907886 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:20.910074 14734 consensus_meta.cc:287] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.543s user 0.009s sys 0.001s
I20260808 20:14:20.826982 14788 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:20.862264 13016 leader_election.cc:302] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.864710 12048 raft_consensus.cc:1226] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 1. Current term is 2. Ops: []
W20260808 20:14:20.911751 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 49599.7us
I20260808 20:14:20.912317 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:20.912966 12048 raft_consensus.cc:3037] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:20.913013 14788 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:14:20.913683 14788 raft_consensus.cc:2781] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:20.873955 14680 consensus_queue.cc:1046] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:20.875097 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:20.887387 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:20.887840 14840 raft_consensus.cc:2726] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:20.889256 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:20.889884 14533 consensus_queue.cc:237] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:20.891600 14831 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:20.895000 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:20.896361 14850 consensus_queue.cc:1035] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:20.897032 14766 raft_consensus.cc:2781] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:20.914130 14788 raft_consensus.cc:686] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:20.917928 12102 leader_election.cc:302] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.945322 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:20.884276 14799 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:20.946005 12040 raft_consensus.cc:2454] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:20.916172 12142 raft_consensus.cc:2932] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:20.923230 12039 raft_consensus.cc:1261] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 20:14:20.922855 12147 raft_consensus.cc:2379] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:20.968762 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:20.969424 12147 raft_consensus.cc:2362] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4. Re-sending same reply.
I20260808 20:14:20.970348 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:20.921574 12150 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
W20260808 20:14:20.947554 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 174544us
I20260808 20:14:20.948856 14880 consensus_queue.cc:1046] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260808 20:14:20.949229 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 162019us
I20260808 20:14:20.972018 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:20.971407 12102 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.974650 14788 consensus_queue.cc:237] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:20.954556 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 96739.6us
I20260808 20:14:20.955003 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:20.975997 13078 raft_consensus.cc:2454] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:20.976321 12000 leader_election.cc:302] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.922147 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:20.977200 12001 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:20.955752 14890 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:20.980212 14831 raft_consensus.cc:2726] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:20.946511 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:20.980509 14890 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:20.977348 12000 leader_election.cc:302] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:20.982556 14890 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:20.983002 12000 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:20.983337 14890 raft_consensus.cc:2781] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:20.983505 12248 raft_consensus.cc:1226] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 6. Current term is 7. Ops: []
I20260808 20:14:20.983721 14890 raft_consensus.cc:482] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:20.984129 14890 raft_consensus.cc:3037] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:20.984933 14704 raft_consensus.cc:2781] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:20.985308 14704 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:20.985673 14704 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:20.987064 14684 raft_consensus.cc:1050] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
W20260808 20:14:20.987447 14684 raft_consensus.cc:1054] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995: Unable to promote non-voter e5bd9a4616ce4acfa20dc7a002984a20: Illegal state: Leader has not yet committed an operation in its own term
I20260808 20:14:20.987828 14684 raft_consensus.cc:3032] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:20.988132 14684 raft_consensus.cc:729] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:20.936347 14868 consensus_queue.cc:1046] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.068s
I20260808 20:14:20.972826 14822 raft_consensus.cc:2726] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:20.978555 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:20.979290 12137 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:20.979453 14680 consensus_queue.cc:1035] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 20:14:20.939718 12138 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:20.958135 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:20.971400 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:21.024747 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:20.990979 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:21.025754 13079 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:21.026593 12147 raft_consensus.cc:2454] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:20.991763 14890 raft_consensus.cc:504] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:20.991801 14799 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:20.990280 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:21.011377 12152 raft_consensus.cc:2396] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 20:14:21.009231 12250 raft_consensus.cc:1226] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:21.018699 13015 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:21.029246 12244 raft_consensus.cc:2454] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:21.020995 12040 raft_consensus.cc:1261] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 20:14:21.027082 12001 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:21.027549 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:21.032948 12147 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:21.023252 12046 raft_consensus.cc:1261] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260808 20:14:20.998206 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:21.000536 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:20.991946 14831 raft_consensus.cc:2781] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:21.014988 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:20.992059 12000 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:21.040063 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 48056us
I20260808 20:14:21.004067 14891 raft_consensus.cc:2726] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:21.041388 14891 raft_consensus.cc:2726] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:20.991535 12254 raft_consensus.cc:3032] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:21.041787 12000 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:21.042084 12254 raft_consensus.cc:729] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
W20260808 20:14:20.995003 14851 log.cc:924] Time spent T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.490s user 0.002s sys 0.000s
I20260808 20:14:21.042728 12000 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:21.042935 12254 consensus_queue.cc:260] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:21.029209 12245 raft_consensus.cc:2379] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 5: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:21.045257 12000 leader_election.cc:302] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:21.046298 12150 raft_consensus.cc:2454] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
W20260808 20:14:21.058867 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 43983.6us
I20260808 20:14:21.059208 14902 raft_consensus.cc:2781] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:21.066573 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:21.067502 12140 raft_consensus.cc:2379] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:21.068825 12147 raft_consensus.cc:2454] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 7.
I20260808 20:14:21.068945 11994 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:21.069517 12150 raft_consensus.cc:1226] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260808 20:14:21.029608 14822 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:21.005673 14766 raft_consensus.cc:686] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
W20260808 20:14:21.095904 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 55341.4us
I20260808 20:14:21.006289 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:21.094545 14614 raft_consensus.cc:2726] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:21.016773 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:21.010537 14850 raft_consensus.cc:482] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:21.096949 12140 raft_consensus.cc:1226] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260808 20:14:21.067829 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:21.011996 14800 consensus_queue.cc:1046] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.081s
I20260808 20:14:21.015623 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:21.100592 12206 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:21.040643 14809 raft_consensus.cc:2781] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:21.059576 14891 raft_consensus.cc:2781] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election won for term 5
W20260808 20:14:21.124539 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 58187.5us
W20260808 20:14:21.096431 14873 log.cc:924] Time spent T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260808 20:14:21.017102 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:21.016556 14680 consensus_queue.cc:1035] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.079s
I20260808 20:14:21.039469 14831 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:21.011490 14734 consensus_queue.cc:1046] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.085s
I20260808 20:14:21.115629 12140 raft_consensus.cc:1261] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 20:14:21.029832 14704 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:21.125973 11994 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:21.137552 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 89148.1us
I20260808 20:14:21.139110 12140 raft_consensus.cc:1261] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 20:14:21.140341 12140 raft_consensus.cc:1261] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 20:14:21.128012 14793 raft_consensus.cc:2781] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:20.989064 14789 consensus_queue.cc:1035] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.107s
W20260808 20:14:21.129428 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 84254.2us
I20260808 20:14:21.068216 14799 consensus_queue.cc:1035] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20260808 20:14:21.066809 14837 consensus_queue.cc:1035] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20260808 20:14:21.124614 14905 raft_consensus.cc:2726] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:21.115607 14795 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:21.145629 12000 leader_election.cc:302] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:21.115542 14809 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:21.135385 14831 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:21.139299 12141 raft_consensus.cc:2379] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 5: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:21.149488 12241 raft_consensus.cc:2379] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 3: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:21.170872 13016 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:21.172403 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 45096us
I20260808 20:14:21.098228 14822 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:20.988864 14684 consensus_queue.cc:260] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:21.187589 12101 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:21.161489 14880 raft_consensus.cc:2781] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election won for term 7
I20260808 20:14:21.178287 14903 raft_consensus.cc:2781] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:21.177277 14809 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:21.197620 12040 raft_consensus.cc:1226] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:21.116412 14850 raft_consensus.cc:504] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:21.161413 12048 raft_consensus.cc:2454] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:21.178512 14881 raft_consensus.cc:3032] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:21.196834 14760 raft_consensus.cc:3037] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:21.151961 13071 raft_consensus.cc:2932] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:21.234230 12102 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:21.204301 14905 raft_consensus.cc:2726] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:21.177079 14795 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:21.231607 14684 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:21.228761 14890 leader_election.cc:288] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:21.251850 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 106296us
I20260808 20:14:21.240211 14732 consensus_queue.cc:1046] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:21.239123 14900 raft_consensus.cc:2781] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:21.178598 14766 consensus_queue.cc:237] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:21.254377 12000 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:21.225957 14787 consensus_queue.cc:1046] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260808 20:14:21.240784 12138 raft_consensus.cc:1226] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260808 20:14:21.245591 14826 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:21.207937 14903 raft_consensus.cc:686] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:21.266749 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:21.268363 12046 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:21.269133 12000 leader_election.cc:302] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:21.271014 14911 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:21.268060 14905 raft_consensus.cc:2781] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:21.272233 14905 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
W20260808 20:14:21.272727 14896 log.cc:924] Time spent T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20260808 20:14:21.273293 14787 raft_consensus.cc:3032] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:21.273694 14787 raft_consensus.cc:729] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:21.274184 14400 raft_consensus.cc:3032] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:21.274554 14400 raft_consensus.cc:729] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:21.271487 14911 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:21.277482 14822 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:21.279320 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:21.279541 12144 raft_consensus.cc:1261] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20260808 20:14:21.280148 12144 pending_rounds.cc:76] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9: Aborting all ops after (but not including) 4
I20260808 20:14:21.280522 12144 pending_rounds.cc:97] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260808 20:14:21.257714 14704 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:21.258121 14900 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:21.272636 14905 raft_consensus.cc:3037] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:21.311393 14850 leader_election.cc:288] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:21.287834 12144 raft_consensus.cc:2866] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:21.312940 14910 consensus_queue.cc:1046] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
I20260808 20:14:21.316397 12046 raft_consensus.cc:2454] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:21.336849 12102 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
W20260808 20:14:21.334617 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 65556.3us
I20260808 20:14:21.207880 14880 raft_consensus.cc:686] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 7 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:21.275498 14400 consensus_queue.cc:260] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:21.290401 14733 raft_consensus.cc:482] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:21.274633 14787 consensus_queue.cc:260] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:21.351658 12000 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:21.230754 14881 raft_consensus.cc:729] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:21.340752 14680 raft_consensus.cc:2726] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:21.346262 14919 raft_consensus.cc:2781] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:21.318365 14900 raft_consensus.cc:3037] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:21.312696 14911 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:21.341266 14850 raft_consensus.cc:3032] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:21.370358 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:21.396718 12144 raft_consensus.cc:1261] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 20:14:21.357892 14733 raft_consensus.cc:504] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:21.403429 14733 leader_election.cc:288] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:21.404567 14882 consensus_queue.cc:1035] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:21.406042 14760 raft_consensus.cc:2726] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:21.394290 14911 raft_consensus.cc:2781] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:21.406669 14882 raft_consensus.cc:2726] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:21.345216 14890 consensus_queue.cc:1035] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20260808 20:14:21.406965 14880 consensus_queue.cc:237] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:21.415073 14911 raft_consensus.cc:686] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:21.370024 14680 consensus_queue.cc:1035] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20260808 20:14:21.416280 14890 raft_consensus.cc:482] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:21.367293 14850 raft_consensus.cc:729] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:21.360766 14809 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:21.418902 12043 raft_consensus.cc:1226] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:21.348688 14903 consensus_queue.cc:237] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:21.430867 12146 raft_consensus.cc:1226] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260808 20:14:21.420034 14787 raft_consensus.cc:3037] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:21.420118 13080 raft_consensus.cc:1261] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 20:14:21.368412 14919 raft_consensus.cc:686] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:21.437863 12036 raft_consensus.cc:1261] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 20:14:21.436653 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:21.438273 14919 consensus_queue.cc:237] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:21.423869 14890 raft_consensus.cc:504] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:21.441694 13072 raft_consensus.cc:2454] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:21.459697 12207 leader_election.cc:302] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:21.408335 14881 consensus_queue.cc:260] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:21.481122 14881 raft_consensus.cc:3037] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:14:21.481642 14860 log.cc:924] Time spent T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.978s user 0.003s sys 0.000s
I20260808 20:14:21.481562 14733 raft_consensus.cc:2781] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:21.485000 14733 raft_consensus.cc:482] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:21.485462 14733 raft_consensus.cc:3037] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:21.431859 14795 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:21.493645 11894 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:57500 (request call id 311) took 12872 ms (client timeout 15000 ms). Trace:
I20260808 20:14:21.502673 12137 raft_consensus.cc:1261] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260808 20:14:21.503232 12137 pending_rounds.cc:76] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9: Aborting all ops after (but not including) 1
I20260808 20:14:21.503615 12137 pending_rounds.cc:97] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260808 20:14:21.503684 14818 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:21.504086 12137 raft_consensus.cc:2866] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:21.504396 14818 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:21.506968 14839 consensus_queue.cc:1046] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20260808 20:14:21.513901 14839 raft_consensus.cc:3032] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:21.514394 14839 raft_consensus.cc:729] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:21.458841 14911 consensus_queue.cc:237] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:21.491688 14921 consensus_queue.cc:1035] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260808 20:14:21.448702 14882 consensus_queue.cc:1035] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.048s
I20260808 20:14:21.542475 14922 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:21.543144 14922 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:21.547048 14922 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:21.547823 14786 raft_consensus.cc:3032] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:21.548403 14786 raft_consensus.cc:729] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:21.548938 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:21.550215 12036 raft_consensus.cc:2379] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 2: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
W20260808 20:14:21.494050 11894 rpcz_store.cc:269] 0808 20:14:08.621657 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:08.621868 (+ 211us) service_pool.cc:227] Handling call
0808 20:14:21.493576 (+12871708us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 20:14:19.992951 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 20:14:19.993497 (+ 546us) write_op.cc:432] Acquiring schema lock in shared mode
0808 20:14:19.993531 (+ 34us) write_op.cc:435] Acquired schema lock
0808 20:14:19.993545 (+ 14us) tablet.cc:660] Decoding operations
0808 20:14:19.993940 (+ 395us) write_op.cc:620] Acquiring the partition lock for write op
0808 20:14:19.993997 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0808 20:14:19.994023 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0808 20:14:19.994335 (+ 312us) tablet.cc:699] Row locks acquired
0808 20:14:19.994353 (+ 18us) write_op.cc:260] PREPARE: finished
0808 20:14:19.994509 (+ 156us) write_op.cc:270] Start()
0808 20:14:19.994618 (+ 109us) write_op.cc:276] Timestamp: P: 1786220059994446 usec, L: 0
0808 20:14:19.994637 (+ 19us) op_driver.cc:331] REPLICATION: starting
0808 20:14:19.995272 (+ 635us) log.cc:842] Serialized 907 byte log entry
0808 20:14:20.064743 (+ 69471us) spinlock_profiling.cc:243] Waited 30.9 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa9653945fe 00007fa96e1cba39 00007fa9653b18bb 00007fa9653afcbf 00007fa9653b1e22 00007fa967f5b5bb
0808 20:14:20.721646 (+656903us) op_driver.cc:444] REPLICATION: finished
0808 20:14:20.952074 (+230428us) write_op.cc:301] APPLY: starting
0808 20:14:20.952152 (+ 78us) tablet.cc:1365] starting BulkCheckPresence
0808 20:14:20.952283 (+ 131us) tablet.cc:1368] finished BulkCheckPresence
0808 20:14:20.952306 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 20:14:20.952714 (+ 408us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 20:14:20.952871 (+ 157us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0808 20:14:20.952919 (+ 48us) write_op.cc:312] APPLY: finished
0808 20:14:20.953076 (+ 157us) log.cc:842] Serialized 29 byte log entry
0808 20:14:20.953221 (+ 145us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:14:20.953312 (+ 91us) write_op.cc:454] Released schema lock
0808 20:14:20.953347 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0808 20:14:20.953595 (+ 248us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:14:20.953619 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":36890,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":230161,"apply.run_cpu_time_us":1795,"apply.run_wall_time_us":1796,"num_ops":2,"prepare.queue_time_us":159307,"prepare.run_cpu_time_us":41416,"prepare.run_wall_time_us":423716,"raft.queue_time_us":265569,"raft.run_cpu_time_us":37562,"raft.run_wall_time_us":216363,"replication_time_us":726931,"spinlock_wait_cycles":71013632,"thread_start_us":848524,"threads_started":4,"wal-append.queue_time_us":457725}]]}
I20260808 20:14:21.468508 14850 consensus_queue.cc:260] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:21.589871 12243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1739) took 1111 ms. Trace:
I20260808 20:14:21.589414 14905 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:21.591183 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
W20260808 20:14:21.592993 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 86399us
I20260808 20:14:21.595172 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:21.595952 13072 raft_consensus.cc:2454] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:21.537973 14818 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:21.601357 12001 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:21.590431 12243 rpcz_store.cc:276] 0808 20:14:20.478376 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:20.495154 (+ 16778us) service_pool.cc:227] Handling call
0808 20:14:21.589645 (+1094491us) spinlock_profiling.cc:243] Waited 6.7 ms on lock 0x7b5400375354. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:21.589780 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15405824}
I20260808 20:14:21.647171 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:21.648000 14924 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:21.608261 14808 raft_consensus.cc:2781] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:21.649209 14396 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:21.570358 14793 raft_consensus.cc:686] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:21.648727 14924 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:21.549422 14786 consensus_queue.cc:260] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:21.549968 14839 consensus_queue.cc:260] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:21.687851 14906 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:21.655094 14808 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:21.620587 14809 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:21.604409 14831 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:21.708467 14831 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:21.699759 14786 raft_consensus.cc:3037] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:21.700906 12147 raft_consensus.cc:1261] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260808 20:14:21.711578 12147 pending_rounds.cc:76] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9: Aborting all ops after (but not including) 5
I20260808 20:14:21.711820 12147 pending_rounds.cc:97] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260808 20:14:21.712064 12147 raft_consensus.cc:2866] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:21.705363 14750 raft_consensus.cc:3032] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 20:14:21.714316 14750 raft_consensus.cc:729] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:21.702987 14808 raft_consensus.cc:3037] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:21.713302 12043 raft_consensus.cc:1226] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:21.715613 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:21.720871 14400 raft_consensus.cc:3037] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:21.730880 14733 raft_consensus.cc:504] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:21.741323 13016 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.023s sys 0.032s
W20260808 20:14:21.749440 13016 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.552s user 0.023s sys 0.033s
W20260808 20:14:21.752558 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 582439us
W20260808 20:14:21.760990 12101 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.014s sys 0.041s
W20260808 20:14:21.761389 12101 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.570s user 0.014s sys 0.041s
I20260808 20:14:21.736380 14839 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
W20260808 20:14:21.761787 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 570667us
I20260808 20:14:21.769943 13079 raft_consensus.cc:2454] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:21.697299 14906 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:21.753252 12247 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.658s user 0.023s sys 0.034s
I20260808 20:14:21.771361 12101 leader_election.cc:302] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:21.746150 14738 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:21.814870 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:21.816109 12001 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:21.833272 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:21.833454 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:21.841257 12245 raft_consensus.cc:2379] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:21.742453 14446 consensus_queue.cc:1046] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.070s
I20260808 20:14:21.760133 14407 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:21.810483 12247 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.715s user 0.023s sys 0.034s
I20260808 20:14:21.849249 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:21.818043 14801 raft_consensus.cc:2741] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260808 20:14:21.883662 14905 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:21.846315 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:21.887740 14733 leader_election.cc:288] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:21.858897 14826 consensus_meta.cc:287] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.585s user 0.004s sys 0.008s
I20260808 20:14:21.909886 14809 raft_consensus.cc:2781] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election won for term 3
W20260808 20:14:21.912853 14902 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.774s user 0.031s sys 0.036s
I20260808 20:14:21.842640 14738 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:21.925755 14738 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:21.870860 14407 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:21.927218 12250 raft_consensus.cc:2454] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:21.927662 14890 leader_election.cc:288] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:21.908813 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 134690us
W20260808 20:14:21.938620 13018 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.539s user 0.007s sys 0.050s
W20260808 20:14:21.938992 13018 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.745s user 0.007s sys 0.051s
W20260808 20:14:21.917713 14807 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.777s user 0.017s sys 0.051s
W20260808 20:14:21.916613 14902 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.781s user 0.031s sys 0.036s
I20260808 20:14:21.835698 14750 consensus_queue.cc:260] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:21.928715 14407 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:21.942817 14900 consensus_meta.cc:287] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.565s user 0.005s sys 0.009s
I20260808 20:14:21.920161 14809 raft_consensus.cc:686] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
W20260808 20:14:21.960366 12103 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.013s sys 0.048s
W20260808 20:14:21.960860 12103 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.767s user 0.014s sys 0.048s
W20260808 20:14:21.961303 12206 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.836s user 0.023s sys 0.060s
W20260808 20:14:21.961624 12206 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.836s user 0.023s sys 0.060s
W20260808 20:14:21.962147 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 861595us
I20260808 20:14:21.852833 14793 consensus_queue.cc:237] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:21.966241 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:21.950685 14902 raft_consensus.cc:686] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:21.998934 14902 consensus_queue.cc:237] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:21.947381 14674 raft_consensus.cc:482] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:22.019651 12250 raft_consensus.cc:1261] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260808 20:14:21.948057 14807 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.809s user 0.017s sys 0.051s
I20260808 20:14:22.026443 12101 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:22.030826 12101 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:22.042497 14758 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:22.043340 14758 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:22.044950 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:22.045576 13079 raft_consensus.cc:3037] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:21.964231 14750 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:22.053692 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:22.054613 13076 raft_consensus.cc:2454] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:22.054171 14809 consensus_queue.cc:237] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:22.055877 13076 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEARNER]: Advancing to term 5
I20260808 20:14:22.059073 12207 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:22.016867 14801 raft_consensus.cc:482] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:22.016003 14800 raft_consensus.cc:3032] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:22.017022 14407 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:22.028931 14738 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:22.051425 14733 raft_consensus.cc:482] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:22.104336 14733 raft_consensus.cc:504] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:22.107752 14733 leader_election.cc:288] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:22.030153 14674 raft_consensus.cc:504] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:22.115187 14674 leader_election.cc:288] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:22.068220 14407 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:22.119735 14407 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:22.075834 14906 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:22.077613 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:22.079375 14789 raft_consensus.cc:2726] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:22.125162 12250 raft_consensus.cc:2454] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:22.080866 14904 consensus_queue.cc:1035] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.044s
I20260808 20:14:22.094110 14777 raft_consensus.cc:2781] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:22.101056 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:22.073123 14924 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:22.103724 14684 consensus_meta.cc:287] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.850s user 0.009s sys 0.003s
I20260808 20:14:22.061561 14800 raft_consensus.cc:729] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:22.157495 13074 raft_consensus.cc:2454] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:14:22.158005 14800 consensus_queue.cc:260] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:22.159219 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:22.129554 13014 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:22.128825 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:22.068748 14801 raft_consensus.cc:504] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:22.131582 14789 raft_consensus.cc:482] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:22.134617 14407 raft_consensus.cc:482] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:22.136772 14733 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:22.143846 12254 raft_consensus.cc:3037] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:22.067175 14807 raft_consensus.cc:2781] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:22.159752 14800 raft_consensus.cc:3037] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:22.167639 12250 raft_consensus.cc:2454] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:14:22.168984 14925 raft_consensus.cc:2781] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:22.169745 12207 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:22.172187 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:22.229113 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 59454.7us
I20260808 20:14:22.174886 14850 raft_consensus.cc:3037] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:22.177414 12045 raft_consensus.cc:1226] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:22.003625 14900 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:22.202661 12105 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.998s user 0.011s sys 0.045s
I20260808 20:14:22.219542 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:22.255524 12105 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.051s user 0.011s sys 0.045s
I20260808 20:14:22.148563 14777 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:22.255864 13014 leader_election.cc:302] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:22.148985 13072 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:22.246855 14432 raft_consensus.cc:2781] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:22.249039 14826 raft_consensus.cc:482] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:22.250975 11994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.106s user 0.012s sys 0.043s
I20260808 20:14:22.251726 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260808 20:14:22.331765 11994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.187s user 0.013s sys 0.043s
I20260808 20:14:22.334440 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:22.073779 14738 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:22.255887 12039 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
W20260808 20:14:22.265177 14533 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.252s user 0.010s sys 0.038s
W20260808 20:14:22.274449 12040 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.017s user 0.007s sys 0.041s
I20260808 20:14:22.277621 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:22.365125 12040 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.108s user 0.007s sys 0.041s
W20260808 20:14:22.291639 12102 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.865s user 0.013s sys 0.052s
I20260808 20:14:22.232307 14807 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:22.352774 14920 consensus_queue.cc:1035] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.068s
W20260808 20:14:22.265298 14891 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.103s user 0.009s sys 0.041s
I20260808 20:14:22.242491 11896 catalog_manager.cc:5495] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 4 to 5, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 5 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
W20260808 20:14:22.365839 12102 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.939s user 0.015s sys 0.055s
W20260808 20:14:22.389142 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 962794us
I20260808 20:14:22.275976 14501 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:22.218362 14789 raft_consensus.cc:504] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:22.370262 14533 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.357s user 0.010s sys 0.038s
I20260808 20:14:22.261267 14777 raft_consensus.cc:3037] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:22.359493 14840 raft_consensus.cc:482] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:22.378927 13079 raft_consensus.cc:2454] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:22.339357 14432 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:14:22.274174 13077 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.990s user 0.024s sys 0.024s
I20260808 20:14:22.286247 14674 raft_consensus.cc:2781] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:22.217252 14407 raft_consensus.cc:504] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:22.369489 14921 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:22.227782 14733 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:22.285018 14397 consensus_queue.cc:1035] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.143s
I20260808 20:14:22.261650 14925 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:14:22.415053 13078 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.892s user 0.014s sys 0.056s
W20260808 20:14:22.471377 13078 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.948s user 0.014s sys 0.056s
I20260808 20:14:22.393298 12046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:53014 (request call id 1399) took 1035 ms. Trace:
I20260808 20:14:22.338516 14810 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:22.353880 14826 raft_consensus.cc:504] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:22.427366 11897 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44508 (request call id 192) took 12869 ms (client timeout 15000 ms). Trace:
W20260808 20:14:22.380272 14891 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.223s user 0.009s sys 0.042s
W20260808 20:14:22.440163 12000 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.079s user 0.002s sys 0.049s
W20260808 20:14:22.459086 13077 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.175s user 0.025s sys 0.024s
W20260808 20:14:22.564720 11897 rpcz_store.cc:269] 0808 20:14:09.192371 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:09.192568 (+ 197us) service_pool.cc:227] Handling call
0808 20:14:22.058156 (+12865588us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":5}
W20260808 20:14:22.565272 12000 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.204s user 0.002s sys 0.052s
W20260808 20:14:22.391372 14786 consensus_meta.cc:287] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.681s user 0.006s sys 0.006s
W20260808 20:14:22.429118 12138 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.020s user 0.029s sys 0.045s
I20260808 20:14:22.374296 14807 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:14:22.404526 13075 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.194s user 0.009s sys 0.033s
I20260808 20:14:22.390590 14758 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:22.472303 12046 rpcz_store.cc:276] 0808 20:14:21.347805 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:21.349762 (+ 1957us) service_pool.cc:227] Handling call
0808 20:14:22.382195 (+1032433us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400170cd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825faf0 00007fa9682e2f2a 00007fa96d21fc6e 00007fa966916807 00007fa966916792 00007fa9669166c1 00007fa96691663c 00007fa9669158f2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:22.382670 (+ 475us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2331399552}
I20260808 20:14:22.462941 14674 raft_consensus.cc:482] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:22.462808 14432 raft_consensus.cc:3037] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
W20260808 20:14:22.560061 12140 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.389s user 0.020s sys 0.073s
I20260808 20:14:22.404078 11896 catalog_manager.cc:5495] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 1 to 3, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 3 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
W20260808 20:14:22.565447 13080 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.911s user 0.001s sys 0.017s
I20260808 20:14:22.474769 14925 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:22.465067 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
W20260808 20:14:22.591609 13080 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.937s user 0.002s sys 0.017s
W20260808 20:14:22.559489 14787 consensus_meta.cc:287] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 1.119s user 0.012s sys 0.007s
I20260808 20:14:22.465569 14840 raft_consensus.cc:504] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:22.428159 14501 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:22.590646 12140 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.420s user 0.020s sys 0.073s
W20260808 20:14:22.587610 12138 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.174s user 0.030s sys 0.045s
I20260808 20:14:22.591390 11896 catalog_manager.cc:5495] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 1 to 3, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 3 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
W20260808 20:14:22.589373 13075 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.386s user 0.012s sys 0.041s
I20260808 20:14:22.582819 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:22.596498 14903 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.871s user 0.006s sys 0.034s
I20260808 20:14:22.590359 14674 raft_consensus.cc:3037] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:22.579099 14891 raft_consensus.cc:686] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
W20260808 20:14:22.597551 14839 consensus_meta.cc:287] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.781s user 0.012s sys 0.001s
I20260808 20:14:22.469431 14921 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:22.614910 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:22.595393 14941 raft_consensus.cc:482] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:22.589452 14939 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
W20260808 20:14:22.582904 14911 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.887s user 0.007s sys 0.032s
I20260808 20:14:22.544530 14810 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:22.593120 14920 raft_consensus.cc:482] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:22.621400 14903 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.900s user 0.006s sys 0.034s
I20260808 20:14:22.700855 12248 raft_consensus.cc:2396] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 20:14:22.582556 14740 consensus_queue.cc:1035] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260808 20:14:22.698132 13079 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:22.596194 14924 consensus_queue.cc:1035] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260808 20:14:22.613142 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:22.697813 12243 raft_consensus.cc:2379] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 5: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:22.724670 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:22.861639 12244 raft_consensus.cc:2454] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:22.860126 12250 raft_consensus.cc:2454] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:22.867049 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
W20260808 20:14:22.870000 14956 log.cc:924] Time spent T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.164s user 0.000s sys 0.000s
I20260808 20:14:22.869015 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:22.854009 14941 raft_consensus.cc:504] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:22.874743 14941 leader_election.cc:288] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:22.875881 14903 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:22.876636 14903 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:22.887943 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:22.889129 14959 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:22.893759 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1263) took 1677 ms. Trace:
I20260808 20:14:22.894134 12040 rpcz_store.cc:276] 0808 20:14:21.216334 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:21.216465 (+ 131us) service_pool.cc:227] Handling call
0808 20:14:21.216745 (+ 280us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:21.231828 (+ 15083us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:21.231853 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:21.232276 (+ 423us) log.cc:842] Serialized 26 byte log entry
0808 20:14:22.365541 (+1133265us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:22.365584 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:22.365653 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:22.893539 (+527886us) raft_consensus.cc:1671] finished
0808 20:14:22.893560 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:22.893695 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":16505,"raft.run_cpu_time_us":67,"raft.run_wall_time_us":68,"thread_start_us":1108409,"threads_started":1,"wal-append.queue_time_us":1356554}
W20260808 20:14:22.910454 12251 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.846s user 0.012s sys 0.009s
W20260808 20:14:22.910851 12251 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.847s user 0.012s sys 0.009s
I20260808 20:14:22.844460 13074 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
W20260808 20:14:22.900106 14788 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.962s user 0.024s sys 0.052s
W20260808 20:14:22.707885 14911 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.017s user 0.007s sys 0.033s
I20260808 20:14:22.846583 14760 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
W20260808 20:14:22.908156 14880 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.073s user 0.009s sys 0.027s
I20260808 20:14:22.702927 14801 leader_election.cc:288] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:22.847631 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1271) took 1282 ms. Trace:
I20260808 20:14:22.902724 13076 raft_consensus.cc:2454] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:14:22.921940 13072 raft_consensus.cc:2379] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
W20260808 20:14:22.920996 14788 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.996s user 0.024s sys 0.053s
I20260808 20:14:22.724704 14839 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:22.845127 14920 raft_consensus.cc:504] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:22.858767 14900 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:22.862332 14738 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:14:22.907856 13071 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.466s user 0.013s sys 0.025s
W20260808 20:14:23.105779 13071 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.664s user 0.013s sys 0.026s
I20260808 20:14:22.903199 14903 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:22.899626 14766 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.586s user 0.023s sys 0.054s
W20260808 20:14:23.107575 14766 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.793s user 0.023s sys 0.055s
I20260808 20:14:22.889858 14959 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:22.699119 14683 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.207s user 0.009s sys 0.015s
I20260808 20:14:23.025092 12248 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1751) took 1255 ms. Trace:
I20260808 20:14:22.917122 14929 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:22.626577 14786 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:23.008276 14733 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:14:22.926390 14880 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.091s user 0.009s sys 0.028s
W20260808 20:14:22.918040 14868 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.848s user 0.011s sys 0.009s
I20260808 20:14:23.089339 14760 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:23.102427 14954 log.cc:924] Time spent T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.086s user 0.001s sys 0.001s
I20260808 20:14:23.104987 14920 leader_election.cc:288] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:22.926678 14789 leader_election.cc:288] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:23.087249 14800 consensus_meta.cc:287] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.861s user 0.009s sys 0.001s
I20260808 20:14:22.814836 14939 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:23.113055 12248 rpcz_store.cc:276] 0808 20:14:21.769661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:21.769815 (+ 154us) service_pool.cc:227] Handling call
0808 20:14:22.925288 (+1155473us) spinlock_profiling.cc:243] Waited 841 ms on lock 0x7b5400166554. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:23.025025 (+ 99737us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1933298560}
W20260808 20:14:23.114769 14958 log.cc:924] Time spent T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.094s user 0.000s sys 0.001s
I20260808 20:14:22.847584 14684 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:23.193464 14684 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:23.196218 14684 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:23.088128 14953 log.cc:924] Time spent T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.196s user 0.000s sys 0.001s
W20260808 20:14:23.098647 14881 consensus_meta.cc:287] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 1.608s user 0.013s sys 0.009s
W20260808 20:14:23.119370 12142 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.982s user 0.031s sys 0.049s
I20260808 20:14:23.173051 14501 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:23.086978 14407 leader_election.cc:288] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:23.014765 14962 raft_consensus.cc:482] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
W20260808 20:14:23.125379 12048 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.775s user 0.052s sys 0.130s
W20260808 20:14:23.111769 14683 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.620s user 0.009s sys 0.015s
W20260808 20:14:23.174952 12001 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.24.4:36901 timed out after 1.536s (SENT) [suppressed 79 similar messages]
W20260808 20:14:23.115288 14945 log.cc:924] Time spent T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.385s user 0.001s sys 0.000s
I20260808 20:14:23.118539 14929 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:23.112110 14738 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:23.183355 14919 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.580s user 0.016s sys 0.025s
I20260808 20:14:23.283083 14943 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:23.283501 14919 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.696s user 0.016s sys 0.028s
I20260808 20:14:23.203449 14840 leader_election.cc:288] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:23.185493 14959 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:23.116742 14826 leader_election.cc:288] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:23.125854 13073 raft_consensus.cc:1226] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4. Ops: []
W20260808 20:14:23.177033 14868 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.111s user 0.012s sys 0.010s
W20260808 20:14:23.203033 12142 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.066s user 0.032s sys 0.049s
W20260808 20:14:23.097965 14396 consensus_meta.cc:287] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 1.425s user 0.004s sys 0.012s
W20260808 20:14:23.206130 12102 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.800s user 0.015s sys 0.018s
W20260808 20:14:23.308562 12102 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.903s user 0.015s sys 0.018s
I20260808 20:14:23.109831 14839 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:23.006953 14891 consensus_queue.cc:237] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:23.176994 12249 raft_consensus.cc:1226] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 7 LEADER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 6. Current term is 7. Ops: []
W20260808 20:14:23.213008 12036 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.366s user 0.021s sys 0.107s
I20260808 20:14:23.125963 14786 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:23.357197 12249 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 287) took 1761 ms. Trace:
I20260808 20:14:23.357622 12249 rpcz_store.cc:276] 0808 20:14:21.595883 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:21.598819 (+ 2936us) service_pool.cc:227] Handling call
0808 20:14:21.599241 (+ 422us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:23.321350 (+1722109us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:23.356965 (+ 35615us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b540007e754. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:23.357131 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3627998464}
I20260808 20:14:23.184052 13071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 375) took 2152 ms. Trace:
W20260808 20:14:23.256361 12048 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.897s user 0.052s sys 0.130s
I20260808 20:14:23.288084 13077 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 371) took 2314 ms. Trace:
W20260808 20:14:23.283943 14967 log.cc:924] Time spent T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.103s user 0.000s sys 0.000s
I20260808 20:14:23.283680 14943 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:23.283221 14738 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:23.395325 14738 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:23.409703 12137 raft_consensus.cc:1226] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:23.411078 13073 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53296 (request call id 613) took 2498 ms. Trace:
I20260808 20:14:23.411530 13073 rpcz_store.cc:276] 0808 20:14:20.912842 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:20.938114 (+ 25272us) service_pool.cc:227] Handling call
0808 20:14:20.979850 (+ 41736us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:23.313606 (+2333756us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:23.410784 (+ 97178us) spinlock_profiling.cc:243] Waited 2.13 s on lock 0x7b540073f5d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:23.411001 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4898075008}
I20260808 20:14:23.412371 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:23.413046 12146 raft_consensus.cc:2454] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:23.413986 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:23.414587 12146 raft_consensus.cc:2454] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:23.415350 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:23.416020 12146 raft_consensus.cc:2379] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:23.416857 12146 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
W20260808 20:14:23.417568 13014 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.038s user 0.044s sys 0.037s
W20260808 20:14:23.417900 13014 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.038s user 0.044s sys 0.037s
I20260808 20:14:23.418149 12247 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1644) took 2395 ms. Trace:
I20260808 20:14:23.418601 12247 rpcz_store.cc:276] 0808 20:14:21.022779 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:21.022887 (+ 108us) service_pool.cc:227] Handling call
0808 20:14:21.024587 (+ 1700us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:21.094813 (+ 70226us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:21.094851 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:21.095341 (+ 490us) log.cc:842] Serialized 26 byte log entry
0808 20:14:21.918168 (+822827us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:21.920877 (+ 2709us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:21.933117 (+ 12240us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:23.417891 (+1484774us) raft_consensus.cc:1671] finished
0808 20:14:23.417912 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:23.418060 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36253,"raft.queue_time_us":59011,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":28988,"thread_start_us":761631,"threads_started":1,"wal-append.queue_time_us":1504553}
W20260808 20:14:23.421448 12206 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.083s user 0.018s sys 0.032s
W20260808 20:14:23.421769 12206 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.084s user 0.018s sys 0.032s
I20260808 20:14:23.421928 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:23.422667 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:23.356928 12036 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.500s user 0.022s sys 0.107s
I20260808 20:14:23.284581 14921 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:23.423281 12248 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
W20260808 20:14:23.368930 14925 consensus_meta.cc:287] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.758s user 0.005s sys 0.005s
I20260808 20:14:23.278355 14684 raft_consensus.cc:482] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:23.256522 14962 raft_consensus.cc:504] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:23.319722 14957 log.cc:924] Time spent T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.219s user 0.000s sys 0.001s
I20260808 20:14:23.349347 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:23.016450 12041 rpcz_store.cc:276] 0808 20:14:21.565666 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:21.568953 (+ 3287us) service_pool.cc:227] Handling call
0808 20:14:21.586314 (+ 17361us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:22.257358 (+671044us) raft_consensus.cc:1520] Early marking committed up to index 3
0808 20:14:22.260973 (+ 3615us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:22.285250 (+ 24277us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:22.333676 (+ 48426us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:22.847355 (+513679us) spinlock_profiling.cc:243] Waited 594 ms on lock 0x7b5400170cd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:22.847403 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:22.847558 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":29031,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":17711,"spinlock_wait_cycles":1366451968}
I20260808 20:14:23.360877 13077 rpcz_store.cc:276] 0808 20:14:20.973560 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:20.973875 (+ 315us) service_pool.cc:227] Handling call
0808 20:14:21.283131 (+309256us) spinlock_profiling.cc:243] Waited 3.35 ms on lock 0x7b5800005198. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:21.283319 (+ 188us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:21.283647 (+ 328us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:21.283673 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:21.284084 (+ 411us) log.cc:842] Serialized 26 byte log entry
0808 20:14:22.564950 (+1280866us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:22.564996 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:22.565072 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:23.287797 (+722725us) raft_consensus.cc:1671] finished
0808 20:14:23.287817 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:23.288018 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1268,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":11982,"spinlock_wait_cycles":7707520,"thread_start_us":1280520,"threads_started":1,"wal-append.queue_time_us":1336675}
I20260808 20:14:23.313361 13074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 617) took 1056 ms. Trace:
I20260808 20:14:23.374626 14943 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:23.386775 12251 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1757) took 1547 ms. Trace:
I20260808 20:14:23.320668 14959 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:23.390877 14750 consensus_meta.cc:287] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 1.330s user 0.008s sys 0.002s
W20260808 20:14:23.393308 14978 log.cc:924] Time spent T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260808 20:14:23.313717 14810 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:23.237785 12241 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:23.464737 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:23.358897 14789 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:23.359123 14900 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
W20260808 20:14:23.496251 12103 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.141s user 0.036s sys 0.149s
I20260808 20:14:23.367432 14396 raft_consensus.cc:482] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:23.521279 12152 raft_consensus.cc:2454] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:14:23.425490 14786 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:23.463244 14766 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:23.806337 11995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.963s user 0.019s sys 0.079s
I20260808 20:14:23.483986 14927 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
W20260808 20:14:23.587128 12047 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.112s user 0.009s sys 0.042s
W20260808 20:14:24.118983 12001 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.863s user 0.011s sys 0.048s
W20260808 20:14:23.761480 12000 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.010s user 0.102s sys 0.229s
W20260808 20:14:23.801386 12140 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.175s user 0.038s sys 0.059s
W20260808 20:14:25.037684 12103 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.700s user 0.084s sys 0.259s
I20260808 20:14:25.037292 12152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 253) took 1688 ms. Trace:
W20260808 20:14:23.771130 13016 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.124s user 0.033s sys 0.061s
W20260808 20:14:23.604068 13078 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.128s user 0.010s sys 0.041s
W20260808 20:14:23.779670 13015 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.273s user 0.022s sys 0.025s
W20260808 20:14:23.755488 11896 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.046s user 0.019s sys 0.025s
W20260808 20:14:23.869102 12101 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.819s user 0.009s sys 0.024s
W20260808 20:14:23.825755 12205 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.467s user 0.023s sys 0.009s
W20260808 20:14:23.815546 12245 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.431s user 0.002s sys 0.031s
W20260808 20:14:23.855052 12105 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.593s user 0.016s sys 0.016s
W20260808 20:14:24.154204 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 505418us
W20260808 20:14:24.229866 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 240663us
W20260808 20:14:25.056470 12205 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.697s user 0.029s sys 0.028s
W20260808 20:14:25.051903 12000 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.300s user 0.151s sys 0.368s
W20260808 20:14:25.085218 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 2.72594e+06us
W20260808 20:14:24.454945 14674 consensus_meta.cc:287] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 1.751s user 0.002s sys 0.008s
W20260808 20:14:25.055780 13078 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.580s user 0.022s sys 0.067s
I20260808 20:14:24.430414 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:23.542007 14818 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
W20260808 20:14:23.572355 14432 consensus_meta.cc:287] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.982s user 0.008s sys 0.004s
W20260808 20:14:25.055974 13015 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.589s user 0.022s sys 0.051s
W20260808 20:14:25.000502 12147 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.849s user 0.000s sys 0.014s
W20260808 20:14:25.096633 12244 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53030 (request call id 303) took 1547 ms (client timeout 1589 ms). Trace:
W20260808 20:14:25.056332 12101 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.006s user 0.021s sys 0.036s
I20260808 20:14:23.809477 13075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53296 (request call id 615) took 2780 ms. Trace:
W20260808 20:14:25.088025 12105 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.827s user 0.029s sys 0.029s
W20260808 20:14:24.303071 12142 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.946s user 0.001s sys 0.008s
W20260808 20:14:25.061281 12245 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.673s user 0.009s sys 0.048s
W20260808 20:14:25.099716 12142 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.743s user 0.003s sys 0.017s
W20260808 20:14:25.052042 12140 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.433s user 0.092s sys 0.158s
W20260808 20:14:24.470103 12254 consensus_meta.cc:287] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 2.221s user 0.006s sys 0.006s
I20260808 20:14:24.308264 15007 raft_consensus.cc:482] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:25.059455 12048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1264) took 3818 ms. Trace:
W20260808 20:14:23.805932 14809 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.205s user 0.008s sys 0.089s
W20260808 20:14:23.820098 14793 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.641s user 0.000s sys 0.034s
W20260808 20:14:24.188542 12138 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.313s user 0.005s sys 0.036s
W20260808 20:14:23.554217 14808 consensus_meta.cc:287] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 1.814s user 0.006s sys 0.007s
W20260808 20:14:25.088249 12047 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.614s user 0.020s sys 0.069s
I20260808 20:14:23.527073 14839 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:25.049779 14396 raft_consensus.cc:504] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.048089 12248 raft_consensus.cc:2396] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for term 3 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
W20260808 20:14:24.583592 14807 consensus_meta.cc:287] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 1.984s user 0.007s sys 0.006s
W20260808 20:14:24.232579 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 818960us
I20260808 20:14:25.090628 14900 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:25.041715 11994 connection.cc:232] RPC call timeout handler was delayed by -1.3462670259177685s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1514s for a response
W20260808 20:14:25.056059 13016 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.409s user 0.083s sys 0.150s
W20260808 20:14:24.300786 14683 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.989s user 0.006s sys 0.003s
W20260808 20:14:24.391394 14948 log.cc:924] Time spent T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.611s user 0.000s sys 0.001s
W20260808 20:14:25.043887 14891 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.005s sys 0.005s
W20260808 20:14:25.168455 14891 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.716s user 0.007s sys 0.005s
I20260808 20:14:25.168080 14808 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.056370 12152 rpcz_store.cc:276] 0808 20:14:23.348970 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:23.349088 (+ 118us) service_pool.cc:227] Handling call
0808 20:14:25.037226 (+1688138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:25.169065 14891 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:25.169674 14891 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.170382 14839 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
W20260808 20:14:23.754734 14902 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.184s user 0.028s sys 0.032s
I20260808 20:14:25.170873 14839 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:25.171437 14902 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.611s user 0.048s sys 0.042s
I20260808 20:14:24.112105 14942 raft_consensus.cc:482] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
W20260808 20:14:24.496011 14850 consensus_meta.cc:287] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 2.241s user 0.011s sys 0.001s
W20260808 20:14:24.390162 14971 log.cc:924] Time spent T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.561s user 0.000s sys 0.001s
W20260808 20:14:23.782135 14400 consensus_meta.cc:287] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 2.061s user 0.014s sys 0.000s
W20260808 20:14:25.162119 12138 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.287s user 0.015s sys 0.046s
W20260808 20:14:24.463644 14777 consensus_meta.cc:287] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 1.988s user 0.011s sys 0.000s
W20260808 20:14:25.096249 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 2.69366e+06us
I20260808 20:14:25.058059 14959 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:25.056685 11896 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.348s user 0.019s sys 0.053s
W20260808 20:14:25.097297 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 3.0469e+06us
I20260808 20:14:25.058741 14927 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:23.804673 12247 raft_consensus.cc:1261] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 20:14:25.130309 14986 raft_consensus.cc:482] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
W20260808 20:14:25.097126 12244 rpcz_store.cc:269] 0808 20:14:23.548939 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:24.310843 (+761904us) service_pool.cc:227] Handling call
0808 20:14:25.096569 (+785726us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:25.095017 14786 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:23.575982 14826 raft_consensus.cc:482] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:24.433283 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 380) took 2817 ms. Trace:
I20260808 20:14:24.428015 14995 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:25.079114 15008 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:23.541538 14903 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:25.035571 14760 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:23.587742 14951 raft_consensus.cc:482] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:24.666182 15001 raft_consensus.cc:482] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:23.495386 14925 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.057142 14810 raft_consensus.cc:482] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:23.489437 14840 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:24.119232 14987 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:25.047343 14962 leader_election.cc:288] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.112062 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:25.061621 15003 raft_consensus.cc:482] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
W20260808 20:14:24.306384 14969 log.cc:924] Time spent T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.784s user 0.000s sys 0.000s
I20260808 20:14:25.048868 14789 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:25.161581 12248 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1657) took 1720 ms. Trace:
W20260808 20:14:25.163681 11994 connection.cc:232] RPC call timeout handler was delayed by -1.3264670260250568s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15360000000000001s for a response
I20260808 20:14:23.541812 13074 rpcz_store.cc:276] 0808 20:14:22.257435 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:22.393174 (+135739us) service_pool.cc:227] Handling call
0808 20:14:23.287537 (+894363us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b54005bed54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:23.313285 (+ 25748us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":247470976}
I20260808 20:14:25.097092 14943 raft_consensus.cc:482] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:23.523525 14684 raft_consensus.cc:504] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:25.096939 14432 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:25.116647 14809 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.522s user 0.052s sys 0.180s
I20260808 20:14:25.175381 14850 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:25.173750 14942 raft_consensus.cc:504] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:24.389042 14976 log.cc:924] Time spent T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 1.020s user 0.000s sys 0.001s
I20260808 20:14:25.181043 14777 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:25.161945 14793 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.983s user 0.012s sys 0.052s
I20260808 20:14:25.054895 14766 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.237519 14951 raft_consensus.cc:504] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.263300 14766 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:25.263461 14951 leader_election.cc:288] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:25.166432 14683 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.862s user 0.008s sys 0.013s
I20260808 20:14:25.248015 14943 raft_consensus.cc:504] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.265565 14683 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:25.265769 14943 leader_election.cc:288] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:24.094043 14929 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.108748 12254 raft_consensus.cc:2454] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:25.127985 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:25.168982 14750 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:25.048153 12251 rpcz_store.cc:276] 0808 20:14:21.839777 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:22.036616 (+196839us) service_pool.cc:227] Handling call
0808 20:14:22.037187 (+ 571us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:22.037572 (+ 385us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:22.037597 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:22.058101 (+ 20504us) log.cc:842] Serialized 548 byte log entry
0808 20:14:22.911266 (+853165us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:14:22.911304 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:22.911364 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:23.386568 (+475204us) raft_consensus.cc:1671] finished
0808 20:14:23.386589 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:23.386715 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":847142,"threads_started":1,"wal-append.queue_time_us":1293903,"wal-append.run_cpu_time_us":1802,"wal-append.run_wall_time_us":1026093}
W20260808 20:14:25.097304 12147 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.946s user 0.000s sys 0.016s
I20260808 20:14:23.855062 14939 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.096374 14818 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:25.097613 14674 raft_consensus.cc:504] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.180776 14732 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:25.186013 12249 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 299) took 1767 ms. Trace:
I20260808 20:14:25.188127 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:25.273301 14817 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260808 20:14:25.194108 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:25.277607 12244 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
W20260808 20:14:25.278069 12244 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53030 (request call id 305) took 1669 ms (client timeout 1998 ms). Trace:
I20260808 20:14:25.180855 14400 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:25.234329 12146 raft_consensus.cc:2454] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:25.281278 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:23.361147 13071 rpcz_store.cc:276] 0808 20:14:21.031345 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:21.036580 (+ 5235us) service_pool.cc:227] Handling call
0808 20:14:21.125732 (+ 89152us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:21.141732 (+ 16000us) raft_consensus.cc:1520] Early marking committed up to index 3
0808 20:14:21.438827 (+297095us) log.cc:842] Serialized 14 byte log entry
0808 20:14:23.120359 (+1681532us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:23.173567 (+ 53208us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:23.180351 (+ 6784us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:23.183417 (+ 3066us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:23.183867 (+ 450us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4717,"raft.queue_time_us":10911,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":55114,"thread_start_us":1667045,"threads_started":1,"wal-append.queue_time_us":1920885}
I20260808 20:14:25.173017 14807 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.117133 15007 raft_consensus.cc:504] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:25.239619 11994 connection.cc:232] RPC call timeout handler was delayed by -1.1554670259356499s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1726s for a response
I20260808 20:14:25.240530 12248 rpcz_store.cc:276] 0808 20:14:23.441256 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:23.464490 (+ 23234us) service_pool.cc:227] Handling call
0808 20:14:25.161520 (+1697030us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:25.245137 13079 raft_consensus.cc:1226] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5. Ops: []
W20260808 20:14:25.247977 11995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.404s user 0.070s sys 0.259s
I20260808 20:14:25.201311 14839 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:25.193228 12247 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1653) took 1755 ms. Trace:
I20260808 20:14:25.192698 14786 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:25.194026 14903 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.260483 14927 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.261445 14396 leader_election.cc:288] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:25.241624 12241 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1752) took 3455 ms (client timeout 1999 ms). Trace:
I20260808 20:14:25.268312 14929 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:25.191672 14826 raft_consensus.cc:504] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:25.273147 12254 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1744) took 4500 ms (client timeout 2393 ms). Trace:
I20260808 20:14:25.277195 12152 raft_consensus.cc:2379] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
W20260808 20:14:25.278442 12244 rpcz_store.cc:269] 0808 20:14:23.608378 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:25.185381 (+1577003us) service_pool.cc:227] Handling call
0808 20:14:25.278017 (+ 92636us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:25.278095 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:25.276773 12249 rpcz_store.cc:276] 0808 20:14:23.418947 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:23.419100 (+ 153us) service_pool.cc:227] Handling call
0808 20:14:23.419380 (+ 280us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:25.177877 (+1758497us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:25.177905 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:25.177945 (+ 40us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:25.177986 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:25.185775 (+ 7789us) spinlock_profiling.cc:243] Waited 1.76 s on lock 0x7b5400142b54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:25.185820 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:25.185949 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4043746560}
W20260808 20:14:25.299376 12251 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1659) took 1781 ms (client timeout 2131 ms). Trace:
I20260808 20:14:25.299454 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:25.299714 13014 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:25.194229 15008 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:25.239413 14810 raft_consensus.cc:504] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.267683 14891 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.286711 11994 leader_election.cc:302] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:25.194344 14995 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.239020 14840 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.289654 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:25.316303 12248 raft_consensus.cc:2454] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:25.317054 12248 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1662) took 1102 ms. Trace:
I20260808 20:14:25.317162 11994 leader_election.cc:302] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:25.317435 12248 rpcz_store.cc:276] 0808 20:14:24.214290 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:25.289408 (+1075118us) service_pool.cc:227] Handling call
0808 20:14:25.316998 (+ 27590us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:25.239320 14987 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:25.253839 15011 raft_consensus.cc:482] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:25.276271 14818 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.252017 14850 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:25.276489 14674 leader_election.cc:288] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.295979 14929 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.272349 15005 consensus_queue.cc:1046] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.198s
I20260808 20:14:25.297926 13080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53290 (request call id 467) took 3678 ms. Trace:
W20260808 20:14:25.299945 12251 rpcz_store.cc:269] 0808 20:14:23.518514 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:25.277825 (+1759311us) service_pool.cc:227] Handling call
0808 20:14:25.299317 (+ 21492us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:25.193185 14986 raft_consensus.cc:504] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:25.295889 12243 raft_consensus.cc:2354] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 4.
I20260808 20:14:25.303839 13079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53290 (request call id 471) took 1784 ms. Trace:
I20260808 20:14:25.347721 13079 rpcz_store.cc:276] 0808 20:14:23.519416 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:24.111397 (+591981us) service_pool.cc:227] Handling call
0808 20:14:24.111822 (+ 425us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:25.287198 (+1175376us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:25.303601 (+ 16403us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b5400353c54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:25.303781 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2603462016}
I20260808 20:14:25.304680 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:25.349342 12152 raft_consensus.cc:3037] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:25.280404 14906 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:25.243378 15001 raft_consensus.cc:504] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.289674 12247 rpcz_store.cc:276] 0808 20:14:23.438334 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:23.465545 (+ 27211us) service_pool.cc:227] Handling call
0808 20:14:23.551831 (+ 86286us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:25.182192 (+1630361us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:25.189541 (+ 7349us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:25.306814 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:25.307349 14962 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:25.308073 12150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1543) took 1924 ms. Trace:
I20260808 20:14:25.389030 12150 rpcz_store.cc:276] 0808 20:14:23.384042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:23.384464 (+ 422us) service_pool.cc:227] Handling call
0808 20:14:23.384748 (+ 284us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:24.149588 (+764840us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:24.149620 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:24.150098 (+ 478us) log.cc:842] Serialized 26 byte log entry
0808 20:14:24.279592 (+129494us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:24.279634 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:24.279711 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:25.307838 (+1028127us) raft_consensus.cc:1671] finished
0808 20:14:25.307861 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:25.307994 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":98268,"raft.queue_time_us":764791,"raft.run_cpu_time_us":58,"raft.run_wall_time_us":59,"thread_start_us":129086,"threads_started":1,"wal-append.queue_time_us":1138704}
I20260808 20:14:25.308547 14902 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:25.303084 14809 raft_consensus.cc:482] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:25.286701 14683 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:25.309363 14951 raft_consensus.cc:482] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:25.311522 14903 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.312093 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:25.246945 15003 raft_consensus.cc:504] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.282274 14400 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:25.289788 14793 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:25.250759 14808 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.291297 14839 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:25.322078 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:25.276646 14732 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.276363 14750 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.315148 14766 raft_consensus.cc:2781] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:25.337090 13080 rpcz_store.cc:276] 0808 20:14:21.619510 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:21.620142 (+ 632us) service_pool.cc:227] Handling call
0808 20:14:21.653320 (+ 33178us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:21.653670 (+ 350us) raft_consensus.cc:1520] Early marking committed up to index 4
0808 20:14:21.653698 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:21.654167 (+ 469us) log.cc:842] Serialized 26 byte log entry
0808 20:14:22.596339 (+942172us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:22.596378 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:22.596440 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:25.297720 (+2701280us) raft_consensus.cc:1671] finished
0808 20:14:25.297742 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:25.297860 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":75,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":116,"thread_start_us":941782,"threads_started":1,"wal-append.queue_time_us":1624690}
I20260808 20:14:25.362838 12251 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1658) took 1921 ms. Trace:
I20260808 20:14:25.466876 12251 rpcz_store.cc:276] 0808 20:14:23.441534 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:25.361928 (+1920394us) service_pool.cc:227] Handling call
0808 20:14:25.362254 (+ 326us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:25.362553 (+ 299us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:25.362579 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:25.362617 (+ 38us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:25.362649 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:25.362696 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:25.362787 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5939,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":26409}
I20260808 20:14:25.468305 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:25.334839 14840 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:14:25.099349 13075 rpcz_store.cc:276] 0808 20:14:21.023032 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:21.023676 (+ 644us) service_pool.cc:227] Handling call
0808 20:14:21.095073 (+ 71397us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:21.095564 (+ 491us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:21.095596 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:21.203559 (+107963us) log.cc:842] Serialized 548 byte log entry
0808 20:14:22.625421 (+1421862us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:14:22.702343 (+ 76922us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:22.726842 (+ 24499us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:23.761755 (+1034913us) raft_consensus.cc:1671] finished
0808 20:14:23.761782 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:23.803327 (+ 41545us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1406382,"threads_started":1,"wal-append.queue_time_us":1707379,"wal-append.run_cpu_time_us":3328,"wal-append.run_wall_time_us":2066329}
I20260808 20:14:25.320320 14891 raft_consensus.cc:482] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
W20260808 20:14:25.292586 12241 rpcz_store.cc:269] 0808 20:14:21.770183 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:21.773836 (+ 3653us) service_pool.cc:227] Handling call
0808 20:14:25.194421 (+3420585us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b5400374e54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:25.225294 (+ 30873us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3033822080}
I20260808 20:14:25.325271 14900 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.335027 14818 raft_consensus.cc:2781] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:25.402545 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:25.401847 14962 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.411274 14986 leader_election.cc:288] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.414029 14995 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:25.395434 15018 log.cc:924] Time spent T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.217s user 0.001s sys 0.000s
I20260808 20:14:25.422241 12249 raft_consensus.cc:2396] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 20:14:25.405215 14902 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.442852 12152 raft_consensus.cc:2454] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:25.407523 14809 raft_consensus.cc:504] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.397082 14925 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:25.478688 12243 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1649) took 3142 ms (client timeout 2173 ms). Trace:
I20260808 20:14:25.457373 12248 raft_consensus.cc:3037] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
W20260808 20:14:25.301007 12254 rpcz_store.cc:269] 0808 20:14:20.773102 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:20.773207 (+ 105us) service_pool.cc:227] Handling call
0808 20:14:20.991415 (+218208us) spinlock_profiling.cc:243] Waited 6.93 ms on lock 0x7b540023a620. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa96826a730 00007fa9682ddeff 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86
0808 20:14:21.507554 (+516139us) spinlock_profiling.cc:243] Waited 71.4 ms on lock 0x7b18002d08a0. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa968315856 00007fa968267f7e 00007fa9682d1367 00007fa9682d2b4e 00007fa9682de228 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c
0808 20:14:21.765370 (+257816us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b58007304b0. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa968255f63 00007fa9682b9aba 00007fa9682d13af 00007fa9682d2b4e 00007fa9682de228 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c
0808 20:14:22.143091 (+377721us) spinlock_profiling.cc:243] Waited 79.7 ms on lock 0x7b540023a620. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa96826840d 00007fa968256050 00007fa9682b9aba 00007fa9682d13af 00007fa9682d2b4e 00007fa9682de228 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1
0808 20:14:25.270906 (+3127815us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":831165184}
I20260808 20:14:25.363934 15011 raft_consensus.cc:504] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.492856 14396 raft_consensus.cc:3032] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:25.476593 14891 raft_consensus.cc:504] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.494318 14396 raft_consensus.cc:729] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:25.495121 14396 consensus_queue.cc:260] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:25.495637 14929 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.414114 14683 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.495790 15008 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.496237 14396 raft_consensus.cc:3037] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:25.497404 14962 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.497928 14732 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.408479 14789 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.371574 14906 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.498620 14891 leader_election.cc:288] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.499370 15011 leader_election.cc:288] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.499840 14906 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.464234 14839 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.438141 14432 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.443317 14684 leader_election.cc:288] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.326907 14959 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.455518 14777 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.455312 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:25.478513 14818 raft_consensus.cc:482] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:25.414481 14951 raft_consensus.cc:504] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.479177 12250 raft_consensus.cc:1226] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:25.458721 14942 leader_election.cc:288] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.472330 14766 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:25.407014 14927 raft_consensus.cc:2781] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:25.489717 12249 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1661) took 1276 ms. Trace:
I20260808 20:14:25.474570 14786 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.478752 14826 leader_election.cc:288] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:25.479871 15007 leader_election.cc:288] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.405907 14807 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.468959 12251 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:25.513448 12250 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1655) took 2073 ms. Trace:
I20260808 20:14:25.513825 12251 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 304) took 1929 ms. Trace:
I20260808 20:14:25.457605 14793 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.514199 12251 rpcz_store.cc:276] 0808 20:14:23.584648 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:25.513100 (+1928452us) service_pool.cc:227] Handling call
0808 20:14:25.513380 (+ 280us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:25.513555 (+ 175us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:25.513579 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:25.513616 (+ 37us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:25.513645 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:25.513689 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:25.513774 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:25.513839 12250 rpcz_store.cc:276] 0808 20:14:23.440118 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:23.440215 (+ 97us) service_pool.cc:227] Handling call
0808 20:14:25.479005 (+2038790us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:25.510009 (+ 31004us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:25.513229 (+ 3220us) spinlock_profiling.cc:243] Waited 2.04 s on lock 0x7b54002ebb50. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d9a09 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:25.513372 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4688167808}
I20260808 20:14:25.506285 14810 leader_election.cc:288] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.508826 14818 raft_consensus.cc:3037] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:25.510051 14766 raft_consensus.cc:3037] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:25.509485 14850 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.518719 14927 raft_consensus.cc:686] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:25.520743 14987 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:25.526317 15001 leader_election.cc:288] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.530030 14400 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.528187 14990 raft_consensus.cc:482] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:25.544230 14839 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.544795 14927 consensus_queue.cc:237] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:25.536209 15003 leader_election.cc:288] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.540591 14809 leader_election.cc:288] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.552579 14951 leader_election.cc:288] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.553548 14793 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.554065 14902 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.539258 14990 raft_consensus.cc:504] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.556733 14990 leader_election.cc:288] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:25.593153 12000 connection.cc:232] RPC call timeout handler was delayed by -1.1567422777879983s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1514s for a response
W20260808 20:14:25.598956 12001 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 2.343s user 0.032s sys 0.139s
I20260808 20:14:25.115870 12048 rpcz_store.cc:276] 0808 20:14:21.241447 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:21.241593 (+ 146us) service_pool.cc:227] Handling call
0808 20:14:21.244370 (+ 2777us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:21.244752 (+ 382us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:21.244783 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:21.258378 (+ 13595us) log.cc:842] Serialized 26 byte log entry
0808 20:14:23.423267 (+2164889us) raft_consensus.cc:1631] Marking committed up to 1
0808 20:14:23.463272 (+ 40005us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:23.489873 (+ 26601us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:23.587158 (+ 97285us) raft_consensus.cc:1671] finished
0808 20:14:23.604230 (+ 17072us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:25.053510 (+1449280us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6229,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":3642,"thread_start_us":2024869,"threads_started":1,"wal-append.queue_time_us":2309219,"wal-append.run_cpu_time_us":5586,"wal-append.run_wall_time_us":1021993}
W20260808 20:14:25.599857 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 2.42482e+06us
I20260808 20:14:25.188342 12036 rpcz_store.cc:276] 0808 20:14:21.615773 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:21.616545 (+ 772us) service_pool.cc:227] Handling call
0808 20:14:21.649192 (+ 32647us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:21.652600 (+ 3408us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:21.657924 (+ 5324us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:21.744984 (+ 87060us) log.cc:842] Serialized 26 byte log entry
0808 20:14:23.490506 (+1745522us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:23.522749 (+ 32243us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:23.542826 (+ 20077us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:24.432962 (+890136us) raft_consensus.cc:1671] finished
0808 20:14:24.432990 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:24.433183 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1970,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":37256,"thread_start_us":1586870,"threads_started":1,"wal-append.queue_time_us":2373015}
I20260808 20:14:25.601437 12047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1759) took 3204 ms. Trace:
I20260808 20:14:25.602000 12047 rpcz_store.cc:276] 0808 20:14:22.396365 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:22.396486 (+ 121us) service_pool.cc:227] Handling call
0808 20:14:22.396790 (+ 304us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:22.397152 (+ 362us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:22.397180 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:22.428597 (+ 31417us) log.cc:842] Serialized 26 byte log entry
0808 20:14:25.162358 (+2733761us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:14:25.599742 (+437384us) log.cc:842] Serialized 14 byte log entry
0808 20:14:25.600413 (+ 671us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:25.600606 (+ 193us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:25.600775 (+ 169us) raft_consensus.cc:1671] finished
0808 20:14:25.600814 (+ 39us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:25.601121 (+ 307us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":398018,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":7487,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":38186,"thread_start_us":3106551,"threads_started":2,"wal-append.queue_time_us":2602165}
W20260808 20:14:25.607929 12103 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.010s sys 0.004s
W20260808 20:14:25.608421 12103 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.519s user 0.010s sys 0.004s
I20260808 20:14:25.465126 12105 leader_election.cc:302] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:25.612486 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 147342us
I20260808 20:14:25.617199 12105 leader_election.cc:302] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:25.478948 12141 raft_consensus.cc:2396] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 2 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
W20260808 20:14:25.623274 12142 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.008s sys 0.006s
W20260808 20:14:25.623770 12142 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.523s user 0.008s sys 0.006s
I20260808 20:14:25.625152 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 236) took 4841 ms. Trace:
I20260808 20:14:25.625536 12142 rpcz_store.cc:276] 0808 20:14:20.783513 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:21.035091 (+251578us) service_pool.cc:227] Handling call
0808 20:14:21.068227 (+ 33136us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:21.068443 (+ 216us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:21.068467 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:21.098027 (+ 29560us) log.cc:842] Serialized 26 byte log entry
0808 20:14:23.308198 (+2210171us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:25.100199 (+1792001us) log.cc:842] Serialized 14 byte log entry
0808 20:14:25.624450 (+524251us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:25.624653 (+ 203us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:25.624742 (+ 89us) raft_consensus.cc:1671] finished
0808 20:14:25.624758 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:25.625028 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":113221,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":74735,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":90,"thread_start_us":4438252,"threads_started":3,"wal-append.queue_time_us":2474631}
W20260808 20:14:25.623293 12140 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.004s sys 0.010s
W20260808 20:14:25.630983 12140 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.530s user 0.005s sys 0.010s
I20260808 20:14:25.632426 12140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1258) took 4619 ms. Trace:
I20260808 20:14:25.633065 12147 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1542) took 2249 ms. Trace:
I20260808 20:14:25.632472 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:25.633605 12147 rpcz_store.cc:276] 0808 20:14:23.383762 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:23.383877 (+ 115us) service_pool.cc:227] Handling call
0808 20:14:23.384182 (+ 305us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 20:14:23.492026 (+107844us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:23.492054 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 20:14:24.151014 (+658960us) log.cc:842] Serialized 52 byte log entry
0808 20:14:25.271672 (+1120658us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:25.271711 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:25.271778 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:25.632519 (+360741us) raft_consensus.cc:1671] finished
0808 20:14:25.632554 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:25.632831 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":98777,"raft.queue_time_us":765138,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":121,"thread_start_us":1120057,"threads_started":1,"wal-append.queue_time_us":947970}
I20260808 20:14:25.633453 12146 raft_consensus.cc:2454] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:25.636031 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:25.637390 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:25.639281 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:25.639878 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:25.641032 12141 raft_consensus.cc:2396] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for term 3 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260808 20:14:25.642060 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:25.640791 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:25.642916 12141 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:14:25.639219 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:25.643944 12146 raft_consensus.cc:2454] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:25.644426 12147 raft_consensus.cc:2379] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:25.645160 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:25.477214 12150 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:14:25.486814 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:25.628181 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:25.444098 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:25.510986 12249 rpcz_store.cc:276] 0808 20:14:24.213807 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:25.299886 (+1086079us) service_pool.cc:227] Handling call
0808 20:14:25.489661 (+189775us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:25.645385 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:25.754719 12147 raft_consensus.cc:3037] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:14:25.755589 12245 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.007s sys 0.007s
W20260808 20:14:25.756172 12245 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s user 0.007s sys 0.007s
I20260808 20:14:25.504638 12248 raft_consensus.cc:2454] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:25.754782 12244 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
W20260808 20:14:25.749527 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 151522us
W20260808 20:14:25.753435 12207 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.617s user 0.002s sys 0.013s
I20260808 20:14:25.516779 12251 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 306) took 1762 ms. Trace:
I20260808 20:14:25.762943 14943 raft_consensus.cc:482] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:25.764998 14400 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:25.765601 14941 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:25.766361 14786 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:25.766069 14400 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:25.769059 12152 raft_consensus.cc:2454] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:25.770195 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:25.764211 14943 raft_consensus.cc:504] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:25.491528 12243 rpcz_store.cc:269] 0808 20:14:22.336170 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:22.866716 (+530546us) service_pool.cc:227] Handling call
0808 20:14:25.478440 (+2611724us) spinlock_profiling.cc:243] Waited 2.42 s on lock 0x7b54002ebb54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:25.478617 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5570026240}
W20260808 20:14:25.694433 13016 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.016s sys 0.003s
I20260808 20:14:25.771149 12152 raft_consensus.cc:2454] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
W20260808 20:14:25.772235 12205 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.615s user 0.006s sys 0.009s
I20260808 20:14:25.773084 15003 consensus_queue.cc:1046] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:25.693039 14891 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:25.774394 12138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1539) took 4418 ms. Trace:
I20260808 20:14:25.707489 12250 raft_consensus.cc:1261] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 20:14:25.783231 12250 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1660) took 1628 ms. Trace:
I20260808 20:14:25.783819 12250 rpcz_store.cc:276] 0808 20:14:24.213135 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:25.705972 (+1492837us) service_pool.cc:227] Handling call
0808 20:14:25.706741 (+ 769us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:25.782929 (+ 76188us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:25.783149 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:25.779906 15030 raft_consensus.cc:2726] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:25.633004 12140 rpcz_store.cc:276] 0808 20:14:21.012922 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:21.144205 (+131283us) service_pool.cc:227] Handling call
0808 20:14:21.144457 (+ 252us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:21.169979 (+ 25522us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:21.170012 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:21.170453 (+ 441us) log.cc:842] Serialized 26 byte log entry
0808 20:14:22.602991 (+1432538us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:25.101173 (+2498182us) log.cc:842] Serialized 14 byte log entry
0808 20:14:25.631881 (+530708us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:25.632018 (+ 137us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:25.632091 (+ 73us) raft_consensus.cc:1671] finished
0808 20:14:25.632102 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:25.632345 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":754203,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":63731,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":4597,"thread_start_us":4468416,"threads_started":4,"wal-append.queue_time_us":2545031,"wal-append.run_cpu_time_us":2040,"wal-append.run_wall_time_us":1095545}
W20260808 20:14:25.785012 12205 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.690s user 0.006s sys 0.010s
I20260808 20:14:25.777009 14777 raft_consensus.cc:2726] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:25.785773 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:25.785591 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 694230us
I20260808 20:14:25.787909 12205 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:25.787560 12140 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
W20260808 20:14:25.788995 12206 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.660s user 0.005s sys 0.009s
W20260808 20:14:25.789438 12206 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.739s user 0.005s sys 0.009s
I20260808 20:14:25.790357 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:25.775713 12101 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.027s sys 0.009s
W20260808 20:14:25.792292 12101 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.606s user 0.028s sys 0.009s
W20260808 20:14:25.779834 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 424153us
I20260808 20:14:25.790457 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:25.767297 14786 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.785988 15003 consensus_queue.cc:1225] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer e5bd9a4616ce4acfa20dc7a002984a20 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260808 20:14:25.793467 15027 raft_consensus.cc:2781] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:25.793633 14919 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:25.794713 15027 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:25.795154 15027 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
W20260808 20:14:25.795517 13015 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.009s sys 0.008s
W20260808 20:14:25.795924 13015 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.616s user 0.009s sys 0.009s
W20260808 20:14:25.796264 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 616795us
I20260808 20:14:25.794554 14919 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.796754 14786 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.695642 14962 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:25.797943 14962 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.785296 14400 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.800400 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:25.801397 11994 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:25.802104 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:25.802681 14786 raft_consensus.cc:2781] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:25.802976 12141 raft_consensus.cc:2454] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:25.803112 14786 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:25.803489 14786 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:25.803958 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:25.692097 14809 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:25.805584 14809 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.808290 14809 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:25.809016 11995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.512s user 0.019s sys 0.005s
I20260808 20:14:25.810030 14951 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:25.794492 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:25.810760 14951 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:25.778818 12207 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.688s user 0.004s sys 0.013s
I20260808 20:14:25.820382 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:25.811441 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:25.825347 14891 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:25.827541 12152 raft_consensus.cc:1226] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:25.828279 14891 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.830020 12150 raft_consensus.cc:1226] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:25.692586 14793 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:25.793645 12142 raft_consensus.cc:2379] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:25.779290 12245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1646) took 3438 ms. Trace:
I20260808 20:14:25.694525 14903 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:25.833932 13016 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.670s user 0.017s sys 0.013s
I20260808 20:14:25.691485 14684 raft_consensus.cc:482] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:25.837105 12150 raft_consensus.cc:1226] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:25.840764 12146 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:25.693554 14902 raft_consensus.cc:482] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
W20260808 20:14:25.838871 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 60777.2us
I20260808 20:14:25.793493 13078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53296 (request call id 614) took 4803 ms. Trace:
I20260808 20:14:25.858296 12144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1551) took 2444 ms. Trace:
I20260808 20:14:25.859736 13014 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:25.858832 12144 rpcz_store.cc:276] 0808 20:14:23.413354 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:23.462400 (+ 49046us) service_pool.cc:227] Handling call
0808 20:14:23.462719 (+ 319us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:25.690711 (+2227992us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:25.690785 (+ 74us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:25.690863 (+ 78us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:25.690923 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:25.848741 (+157818us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b540030c854. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:25.848811 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:25.858037 (+ 9226us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":82359,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":162,"spinlock_wait_cycles":3981302144,"thread_start_us":496037,"threads_started":1}
I20260808 20:14:25.861358 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:25.862022 15040 raft_consensus.cc:2781] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:25.862263 12250 raft_consensus.cc:2454] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:25.865298 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:25.866477 12241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 301) took 2382 ms. Trace:
I20260808 20:14:25.866945 12241 rpcz_store.cc:276] 0808 20:14:23.484401 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:25.486636 (+2002235us) service_pool.cc:227] Handling call
0808 20:14:25.865437 (+378801us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:25.865872 (+ 435us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:25.865907 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:25.865992 (+ 85us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:25.866033 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:25.866119 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:25.866274 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:25.833932 12245 rpcz_store.cc:276] 0808 20:14:22.332938 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:22.333359 (+ 421us) service_pool.cc:227] Handling call
0808 20:14:22.383832 (+ 50473us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:25.099860 (+2716028us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:25.099891 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:25.109149 (+ 9258us) log.cc:842] Serialized 26 byte log entry
0808 20:14:25.756986 (+647837us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:25.760028 (+ 3042us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:25.761685 (+ 1657us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:25.771097 (+ 9412us) raft_consensus.cc:1671] finished
0808 20:14:25.771115 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:25.771328 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":179244,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":10502,"thread_start_us":3343262,"threads_started":2,"wal-append.queue_time_us":442913}
I20260808 20:14:25.869200 15021 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
W20260808 20:14:25.869819 12101 connection.cc:232] RPC call timeout handler was delayed by -2.3333507501520216s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23929999999999998s for a response
W20260808 20:14:25.870560 12101 connection.cc:232] RPC call timeout handler was delayed by -1.612341060070321s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19990000000000002s for a response
I20260808 20:14:25.844614 13078 rpcz_store.cc:276] 0808 20:14:20.989782 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:20.989910 (+ 128us) service_pool.cc:227] Handling call
0808 20:14:20.990216 (+ 306us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:21.009389 (+ 19173us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:21.009416 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:21.521211 (+511795us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b2c00091d88. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa966cbde11 00007fa966ccd442 00007fa968315c11 00007fa9682da58a 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772
0808 20:14:21.521559 (+ 348us) log.cc:842] Serialized 26 byte log entry
0808 20:14:22.471783 (+950224us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:25.092817 (+2621034us) log.cc:842] Serialized 14 byte log entry
0808 20:14:25.526010 (+433193us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:25.526993 (+ 983us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:25.527042 (+ 49us) raft_consensus.cc:1671] finished
0808 20:14:25.757111 (+230069us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:25.793203 (+ 36092us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18842,"prepare.queue_time_us":1832327,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":20886,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":911,"spinlock_wait_cycles":34930432,"thread_start_us":3995678,"threads_started":3,"wal-append.queue_time_us":1856031,"wal-append.run_cpu_time_us":11652,"wal-append.run_wall_time_us":2053262}
I20260808 20:14:25.871950 14959 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:25.872267 15030 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:25.872684 14959 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:25.873082 15030 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.875221 14959 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.875568 15030 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.876128 15030 raft_consensus.cc:482] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:25.871380 12105 connection.cc:232] RPC call timeout handler was delayed by -1.5713290870189667s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19990000000000002s for a response
I20260808 20:14:25.861032 14941 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.821199 14809 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:25.833695 14962 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.833662 14891 raft_consensus.cc:482] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
W20260808 20:14:25.862056 12207 connection.cc:232] RPC call timeout handler was delayed by -1.3567152351606637s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1895s for a response
I20260808 20:14:25.869966 15021 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.866508 12247 raft_consensus.cc:2354] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3.
I20260808 20:14:25.862562 15040 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:25.897061 12140 raft_consensus.cc:2454] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:25.865342 15029 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:25.879258 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:25.905478 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:25.907473 11994 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:25.900786 15045 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:25.908102 14902 raft_consensus.cc:504] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:25.909370 14990 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:25.908967 12251 rpcz_store.cc:276] 0808 20:14:23.754594 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:25.515940 (+1761346us) service_pool.cc:227] Handling call
0808 20:14:25.516208 (+ 268us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:25.516379 (+ 171us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:25.516401 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:25.516436 (+ 35us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:25.516483 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:25.516527 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:25.516725 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:25.877970 14750 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.915292 14990 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.873075 14777 raft_consensus.cc:482] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:25.924839 14777 raft_consensus.cc:504] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:25.903313 14809 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:25.926064 14990 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.926889 14941 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.899127 15040 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:25.918951 14818 raft_consensus.cc:504] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:25.876873 15030 raft_consensus.cc:504] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.923585 14786 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:25.916303 14925 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:25.919301 15001 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:25.918150 15045 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.919781 15029 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:25.927085 14777 leader_election.cc:288] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.946440 14809 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.946151 14925 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.947597 14919 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:25.953123 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 45773.6us
I20260808 20:14:25.955464 12142 raft_consensus.cc:1261] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 2 index: 5. (term mismatch)
I20260808 20:14:25.970301 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:25.957175 12142 pending_rounds.cc:76] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9: Aborting all ops after (but not including) 4
I20260808 20:14:25.971055 12140 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 20:14:25.970950 12244 raft_consensus.cc:2454] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:25.910622 14684 raft_consensus.cc:504] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:25.946378 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:25.955538 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:25.942950 14891 raft_consensus.cc:504] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:25.971553 12142 pending_rounds.cc:97] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9: Aborting uncommitted NO_OP operation due to leader change: 3.5
W20260808 20:14:25.975046 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 85834.7us
I20260808 20:14:25.951608 14951 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.978838 12040 raft_consensus.cc:2454] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:25.929316 14793 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:25.947546 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:25.933362 14903 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:25.984606 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:25.962551 12036 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:14:25.998529 14818 leader_election.cc:288] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.998422 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:25.999912 15051 raft_consensus.cc:2781] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:26.000695 14925 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.971927 14943 leader_election.cc:288] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:26.002334 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:26.002532 14903 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:25.980408 12142 raft_consensus.cc:2866] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:26.013780 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.014717 12043 raft_consensus.cc:2379] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 4: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:26.014904 14951 consensus_queue.cc:1035] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:14:26.011134 15051 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:26.016630 15051 raft_consensus.cc:3037] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:26.017017 14927 raft_consensus.cc:3032] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:26.017458 14927 raft_consensus.cc:729] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:26.018404 14927 consensus_queue.cc:260] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:26.019680 14927 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:26.021098 13080 raft_consensus.cc:3037] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 4
I20260808 20:14:26.022643 12140 raft_consensus.cc:1226] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:26.026162 15045 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:25.991637 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:26.029126 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 48478.2us
I20260808 20:14:25.956761 15001 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:26.057720 12039 raft_consensus.cc:2379] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
W20260808 20:14:26.061522 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 80247.1us
I20260808 20:14:26.061182 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:26.026191 14793 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:26.066133 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:26.080626 15029 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:26.080799 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.081063 14777 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:26.081687 12045 raft_consensus.cc:2454] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:26.081787 14777 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:26.083557 14684 leader_election.cc:288] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:26.084060 14777 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:26.092239 14906 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:26.092669 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.092862 14906 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:26.093776 15021 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:26.094115 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.094867 12038 raft_consensus.cc:2454] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:26.095144 14906 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:26.095502 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.096210 12039 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:14:26.097151 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.097947 12039 raft_consensus.cc:2454] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:26.098804 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:26.099923 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:25.782598 12138 rpcz_store.cc:276] 0808 20:14:21.355615 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:21.355773 (+ 158us) service_pool.cc:227] Handling call
0808 20:14:21.361856 (+ 6083us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:21.407688 (+ 45832us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:21.407722 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:21.408497 (+ 775us) log.cc:842] Serialized 26 byte log entry
0808 20:14:22.620200 (+1211703us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:25.183038 (+2562838us) log.cc:842] Serialized 14 byte log entry
0808 20:14:25.773682 (+590644us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:25.773840 (+ 158us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:25.773931 (+ 91us) raft_consensus.cc:1671] finished
0808 20:14:25.773950 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:25.774184 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":80948,"prepare.queue_time_us":719945,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":31,"raft.queue_time_us":114097,"raft.run_cpu_time_us":511,"raft.run_wall_time_us":13414,"thread_start_us":3999662,"threads_started":3,"wal-append.queue_time_us":1517760,"wal-append.run_cpu_time_us":3731,"wal-append.run_wall_time_us":1178388}
I20260808 20:14:26.058045 12041 raft_consensus.cc:2454] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:14:26.107162 12249 raft_consensus.cc:2379] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 3: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:26.115572 12140 raft_consensus.cc:2379] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:26.080511 14919 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:26.081698 15030 leader_election.cc:288] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:26.115715 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.128319 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.130897 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:26.131374 14766 consensus_meta.cc:287] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.606s user 0.005s sys 0.006s
I20260808 20:14:26.131809 14766 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:26.133962 15029 raft_consensus.cc:482] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:26.134584 15029 raft_consensus.cc:504] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:26.137285 15021 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:26.137923 15021 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:26.138855 14396 raft_consensus.cc:482] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:26.139443 14396 raft_consensus.cc:504] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:26.136620 15029 leader_election.cc:288] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:26.139973 15021 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:26.141036 12249 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:14:26.141785 14396 leader_election.cc:288] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:26.142132 14786 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:26.141799 12249 raft_consensus.cc:595] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:26.138281 14987 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:26.143172 14987 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:26.145247 14919 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:26.145704 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:26.146974 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:26.147882 14919 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:26.123854 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:26.149044 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.150051 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.150215 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.150691 12048 raft_consensus.cc:2454] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:26.151696 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.165118 12251 raft_consensus.cc:3037] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:26.166567 14793 consensus_queue.cc:1035] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:14:26.173473 15027 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:26.175940 15027 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:26.119724 12247 raft_consensus.cc:2454] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:14:26.152333 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.153105 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.177290 12046 raft_consensus.cc:2454] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:26.177398 12047 raft_consensus.cc:2396] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 20:14:26.178262 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.154351 14891 leader_election.cc:288] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:26.154237 12000 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:26.156957 12140 raft_consensus.cc:2454] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:26.158236 14766 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:26.159076 12147 raft_consensus.cc:2454] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:26.096793 14906 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:26.159888 15051 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:26.161387 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.150916 12045 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:26.161795 14818 consensus_queue.cc:1046] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 20:14:26.178359 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.156261 15040 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:26.186836 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:26.189512 12046 raft_consensus.cc:2396] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260808 20:14:26.190864 13080 raft_consensus.cc:2454] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:26.145217 14987 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:26.214221 12038 raft_consensus.cc:1226] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:26.226784 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.240209 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:26.244333 13080 raft_consensus.cc:1261] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 20:14:26.249807 14903 raft_consensus.cc:3032] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:26.255401 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:26.256251 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:26.257802 15051 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:26.260453 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 106314us
I20260808 20:14:26.261981 15040 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:26.227916 14891 raft_consensus.cc:2726] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:26.264175 12040 raft_consensus.cc:1261] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260808 20:14:26.227106 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:26.264678 13075 raft_consensus.cc:2354] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6.
I20260808 20:14:26.265201 12137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1545) took 2854 ms. Trace:
I20260808 20:14:26.220779 12254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 302) took 2680 ms. Trace:
I20260808 20:14:26.269590 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.225466 14906 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:26.274573 12244 raft_consensus.cc:1261] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260808 20:14:26.235802 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:26.283337 14903 raft_consensus.cc:729] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:26.294230 12244 pending_rounds.cc:76] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a: Aborting all ops after (but not including) 2
I20260808 20:14:26.296412 12244 pending_rounds.cc:97] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a: Aborting uncommitted NO_OP operation due to leader change: 4.3
W20260808 20:14:26.286995 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 260482us
I20260808 20:14:26.292584 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:26.297451 13078 raft_consensus.cc:2454] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:14:26.279491 14777 raft_consensus.cc:482] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:26.291143 12000 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:26.296744 12244 raft_consensus.cc:2866] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:26.299697 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.305352 12040 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:14:26.308504 12040 raft_consensus.cc:595] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:26.308947 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:26.304206 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.300470 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.314917 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.316161 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:26.308262 14818 raft_consensus.cc:2781] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:26.317076 14906 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:26.317516 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.318141 12046 raft_consensus.cc:2454] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:26.319193 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:26.321079 12039 raft_consensus.cc:2379] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 5: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:26.321699 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:26.322062 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.303864 15001 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:26.323097 12000 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:26.322994 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:26.330830 12000 leader_election.cc:302] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:26.345211 14951 raft_consensus.cc:2781] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:26.345665 14951 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:26.346072 14951 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:26.301777 14903 consensus_queue.cc:260] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:26.346961 14903 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:26.350533 12039 raft_consensus.cc:1261] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 20:14:26.292929 12137 rpcz_store.cc:276] 0808 20:14:23.411109 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:23.411224 (+ 115us) service_pool.cc:227] Handling call
0808 20:14:23.411583 (+ 359us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:25.660667 (+2249084us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:25.660748 (+ 81us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:25.663256 (+ 2508us) log.cc:842] Serialized 26 byte log entry
0808 20:14:25.759477 (+ 96221us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:25.761549 (+ 2072us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:26.213552 (+452003us) spinlock_profiling.cc:243] Waited 1.84 s on lock 0x7b54002a6054. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:26.225965 (+ 12413us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:26.226798 (+ 833us) raft_consensus.cc:1671] finished
0808 20:14:26.226820 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:26.242537 (+ 15717us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":15258,"raft.run_cpu_time_us":68,"raft.run_wall_time_us":68,"spinlock_wait_cycles":4231542016,"thread_start_us":402276,"threads_started":2,"wal-append.queue_time_us":269718}
I20260808 20:14:26.310698 12138 raft_consensus.cc:2454] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:26.354185 12206 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:26.354460 12000 leader_election.cc:302] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:26.355264 15051 raft_consensus.cc:2781] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260808 20:14:26.771389 12251 consensus_meta.cc:287] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.606s user 0.002s sys 0.007s
I20260808 20:14:26.359076 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:26.927543 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.922032 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:26.470566 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:26.469357 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:26.928613 12251 raft_consensus.cc:2454] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:26.932435 12043 raft_consensus.cc:2396] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260808 20:14:26.389191 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.392699 12040 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:14:26.934388 12040 raft_consensus.cc:595] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:26.357196 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:26.933517 12038 raft_consensus.cc:2454] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
W20260808 20:14:26.936573 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 454007us
I20260808 20:14:26.423115 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.937479 12046 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 1.
I20260808 20:14:26.774398 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:26.904081 12150 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.008s sys 0.029s
W20260808 20:14:26.939064 12150 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s user 0.009s sys 0.029s
I20260808 20:14:26.317107 14818 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:26.935351 12040 raft_consensus.cc:603] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 20:14:26.943341 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1277) took 1632 ms. Trace:
I20260808 20:14:26.487167 11896 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:57488 (request call id 319) took 7443 ms. Trace:
I20260808 20:14:26.944343 12250 raft_consensus.cc:2396] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260808 20:14:26.944885 12245 raft_consensus.cc:2396] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260808 20:14:26.355521 15027 raft_consensus.cc:979] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a: : Instructing follower 78b9263889cd4f1582578a3aa4f644d9 to start an election
W20260808 20:14:26.346257 15076 log.cc:924] Time spent T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.222s user 0.002s sys 0.002s
I20260808 20:14:26.945369 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:26.894865 14951 consensus_meta.cc:287] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.548s user 0.009s sys 0.001s
I20260808 20:14:26.966187 12000 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:26.959568 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.774755 15045 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:26.953876 12244 raft_consensus.cc:2379] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 5: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:26.648238 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:26.971544 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:26.974891 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 292575us
I20260808 20:14:26.970979 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:26.478214 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.966158 12146 rpcz_store.cc:276] 0808 20:14:25.297665 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:25.843869 (+546204us) service_pool.cc:227] Handling call
0808 20:14:25.895642 (+ 51773us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:25.944753 (+ 49111us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:25.944788 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:25.955336 (+ 10548us) log.cc:842] Serialized 26 byte log entry
0808 20:14:26.285378 (+330042us) raft_consensus.cc:1631] Marking committed up to 1
0808 20:14:26.285415 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:26.285483 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:26.374545 (+ 89062us) raft_consensus.cc:1671] finished
0808 20:14:26.374566 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:26.929636 (+555070us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5894,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":1954,"thread_start_us":329609,"threads_started":1,"wal-append.queue_time_us":358500}
I20260808 20:14:26.972575 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.937551 15051 raft_consensus.cc:482] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:26.311441 14777 raft_consensus.cc:504] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:26.355861 14987 raft_consensus.cc:2726] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:26.480182 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.391590 14891 raft_consensus.cc:2781] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:26.969800 12039 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
W20260808 20:14:26.979959 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 206065us
I20260808 20:14:26.967947 14818 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:26.829694 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:26.980589 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.980201 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.978565 12043 raft_consensus.cc:2454] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:14:26.974512 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:26.986192 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:26.988095 12245 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:14:26.970610 14902 leader_election.cc:288] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:26.989996 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.988910 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:26.979112 15051 raft_consensus.cc:3037] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:26.992023 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:26.992846 12243 raft_consensus.cc:2454] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:26.992859 12046 raft_consensus.cc:2454] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:26.982774 12047 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:26.991755 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.978202 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.995434 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.989352 12248 raft_consensus.cc:2454] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:26.997638 12251 raft_consensus.cc:2454] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:26.488739 11889 catalog_manager.cc:5495] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 1 to 3, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4). New cstate: current_term: 3 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:26.994689 12038 raft_consensus.cc:2454] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:26.998728 12048 raft_consensus.cc:2396] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260808 20:14:27.001114 12150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 256) took 1390 ms. Trace:
I20260808 20:14:27.001470 12150 rpcz_store.cc:276] 0808 20:14:25.610486 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:25.919748 (+309262us) service_pool.cc:227] Handling call
0808 20:14:26.281488 (+361740us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:26.281689 (+ 201us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:26.298040 (+ 16351us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:26.298059 (+ 19us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:26.305833 (+ 7774us) log.cc:842] Serialized 26 byte log entry
0808 20:14:26.939516 (+633683us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:26.939552 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:26.939615 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:27.000800 (+ 61185us) raft_consensus.cc:1671] finished
0808 20:14:27.000822 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:27.001053 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2947,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":4612,"spinlock_wait_cycles":23019264,"thread_start_us":617008,"threads_started":1,"wal-append.queue_time_us":669363}
I20260808 20:14:26.994048 12247 raft_consensus.cc:2379] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 3: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:27.003702 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:27.003880 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.985119 14891 raft_consensus.cc:482] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:26.973028 14951 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:27.004944 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:26.977077 15045 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.007893 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:26.995824 15003 raft_consensus.cc:2781] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:27.008930 12038 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:27.006263 12040 raft_consensus.cc:2454] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:27.005753 12036 raft_consensus.cc:2379] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 2: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:27.009383 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:27.010120 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:27.012118 12047 raft_consensus.cc:2454] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:27.008632 15096 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:27.013849 12206 leader_election.cc:302] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:27.014596 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.014678 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:27.006830 14891 raft_consensus.cc:3037] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:27.015671 12040 raft_consensus.cc:1226] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:27.011641 12144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1278) took 1688 ms. Trace:
I20260808 20:14:27.011850 15003 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:26.965690 11896 rpcz_store.cc:276] 0808 20:14:19.043466 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:19.056384 (+ 12918us) service_pool.cc:227] Handling call
0808 20:14:26.486952 (+7430568us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 20:14:24.308699 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 20:14:24.309199 (+ 500us) write_op.cc:432] Acquiring schema lock in shared mode
0808 20:14:24.309229 (+ 30us) write_op.cc:435] Acquired schema lock
0808 20:14:24.309245 (+ 16us) tablet.cc:660] Decoding operations
0808 20:14:25.064711 (+755466us) write_op.cc:620] Acquiring the partition lock for write op
0808 20:14:25.064780 (+ 69us) write_op.cc:641] Partition lock acquired for write op
0808 20:14:25.064813 (+ 33us) tablet.cc:683] Acquiring locks for 3 operations
0808 20:14:25.065205 (+ 392us) tablet.cc:699] Row locks acquired
0808 20:14:25.065226 (+ 21us) write_op.cc:260] PREPARE: finished
0808 20:14:25.065382 (+ 156us) write_op.cc:270] Start()
0808 20:14:25.065521 (+ 139us) write_op.cc:276] Timestamp: P: 1786220065065331 usec, L: 0
0808 20:14:25.065544 (+ 23us) op_driver.cc:331] REPLICATION: starting
0808 20:14:25.066197 (+ 653us) log.cc:842] Serialized 1295 byte log entry
0808 20:14:25.564775 (+498578us) op_driver.cc:444] REPLICATION: finished
0808 20:14:25.565990 (+ 1215us) write_op.cc:301] APPLY: starting
0808 20:14:25.566068 (+ 78us) tablet.cc:1365] starting BulkCheckPresence
0808 20:14:25.566159 (+ 91us) tablet.cc:1368] finished BulkCheckPresence
0808 20:14:25.758526 (+192367us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 20:14:25.774784 (+ 16258us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 20:14:25.775091 (+ 307us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0808 20:14:25.775165 (+ 74us) write_op.cc:312] APPLY: finished
0808 20:14:25.775698 (+ 533us) log.cc:842] Serialized 35 byte log entry
0808 20:14:25.775997 (+ 299us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:14:25.776227 (+ 230us) write_op.cc:454] Released schema lock
0808 20:14:25.776297 (+ 70us) write_op.cc:341] FINISH: Updating metrics
0808 20:14:25.776753 (+ 456us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:14:25.776780 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":240703,"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":988,"apply.run_cpu_time_us":4179,"apply.run_wall_time_us":272028,"num_ops":3,"prepare.queue_time_us":1599582,"prepare.run_cpu_time_us":19814,"prepare.run_wall_time_us":1215061,"raft.queue_time_us":4458,"raft.run_cpu_time_us":1141,"raft.run_wall_time_us":1143,"replication_time_us":499163,"thread_start_us":3123973,"threads_started":4,"wal-append.queue_time_us":481961,"wal-append.run_cpu_time_us":5566,"wal-append.run_wall_time_us":1280072}]]}
I20260808 20:14:27.021775 12206 leader_election.cc:302] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:27.020731 12039 raft_consensus.cc:2454] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:27.020162 12047 pending_rounds.cc:76] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting all ops after (but not including) 3
I20260808 20:14:27.024212 12047 pending_rounds.cc:97] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260808 20:14:27.024410 12047 raft_consensus.cc:2866] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:26.293869 12254 rpcz_store.cc:276] 0808 20:14:23.498826 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:25.500393 (+2001567us) service_pool.cc:227] Handling call
0808 20:14:25.500754 (+ 361us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 20:14:25.501141 (+ 387us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:25.501172 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 20:14:25.833299 (+332127us) log.cc:842] Serialized 52 byte log entry
0808 20:14:25.978944 (+145645us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:25.995868 (+ 16924us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:26.034628 (+ 38760us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:26.178560 (+143932us) raft_consensus.cc:1671] finished
0808 20:14:26.178587 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:26.178757 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":86,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":106,"thread_start_us":100505,"threads_started":1,"wal-append.queue_time_us":224942}
I20260808 20:14:27.029117 11889 catalog_manager.cc:5495] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 4 to 5, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 5 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } }
I20260808 20:14:27.017310 12144 rpcz_store.cc:276] 0808 20:14:25.322999 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:25.860852 (+537853us) service_pool.cc:227] Handling call
0808 20:14:25.927452 (+ 66600us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:25.928004 (+ 552us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:25.928036 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:25.928586 (+ 550us) log.cc:842] Serialized 26 byte log entry
0808 20:14:26.072394 (+143808us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:26.073884 (+ 1490us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:26.103258 (+ 29374us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:26.345541 (+242283us) spinlock_profiling.cc:243] Waited 9.05 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:27.008609 (+663068us) raft_consensus.cc:1671] finished
0808 20:14:27.008647 (+ 38us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:27.011569 (+ 2922us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2331,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":8616,"spinlock_wait_cycles":20818688,"thread_start_us":131647,"threads_started":1,"wal-append.queue_time_us":195450}
I20260808 20:14:27.050182 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.031664 15096 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:27.051055 12251 raft_consensus.cc:2396] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260808 20:14:27.049384 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.050294 11889 catalog_manager.cc:5495] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:27.025831 15003 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:27.053289 15096 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:27.056166 11889 catalog_manager.cc:5495] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:27.059127 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.059847 12241 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:27.060245 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:27.061172 12048 raft_consensus.cc:2396] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 20:14:27.074726 14777 leader_election.cc:288] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:27.077425 15045 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:27.077849 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:27.073333 11889 catalog_manager.cc:5495] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20260808 20:14:27.083351 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:27.083791 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.085732 12043 raft_consensus.cc:2379] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:27.088361 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:27.088146 11889 catalog_manager.cc:5495] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 2 to 4, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4). New cstate: current_term: 4 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:27.088945 12251 raft_consensus.cc:2362] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5. Re-sending same reply.
I20260808 20:14:27.089641 11889 catalog_manager.cc:5495] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 1 to 3, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 3 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:27.091658 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.093207 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.094189 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.094434 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.085693 14951 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:27.097532 12244 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:27.098831 12241 raft_consensus.cc:2396] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260808 20:14:27.096318 11889 catalog_manager.cc:5495] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 3 to 4, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 4 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:27.104257 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.106357 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.106920 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.108157 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:27.108219 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.108886 13079 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:27.107582 12244 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:27.098331 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.112845 11889 catalog_manager.cc:5495] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 2 to 4, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 4 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } }
I20260808 20:14:27.114861 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.116089 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
W20260808 20:14:27.112663 14903 consensus_meta.cc:287] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.765s user 0.012s sys 0.000s
I20260808 20:14:27.120123 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:27.123361 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.123944 13078 raft_consensus.cc:3037] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 3
I20260808 20:14:27.129300 12244 raft_consensus.cc:2362] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5. Re-sending same reply.
I20260808 20:14:27.125434 15027 raft_consensus.cc:2781] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:27.134456 15027 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:27.134691 15104 raft_consensus.cc:2781] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:27.134819 15027 raft_consensus.cc:3037] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:27.135350 15104 raft_consensus.cc:482] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:14:27.135422 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 202501us
I20260808 20:14:27.136034 15104 raft_consensus.cc:3037] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:27.136598 12000 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:27.139206 12000 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:27.139328 14951 raft_consensus.cc:2781] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:27.139775 14951 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:27.140168 14951 raft_consensus.cc:3037] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:27.141460 13077 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
W20260808 20:14:27.145172 12001 leader_election.cc:334] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Timed out: RequestConsensusVote RPC to 127.10.24.4:36901 timed out after 1.536s (SENT)
I20260808 20:14:27.143226 11889 catalog_manager.cc:5495] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 2 to 4, leader changed from e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 4 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:14:27.147415 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.147738 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.148528 13075 raft_consensus.cc:2454] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:14:27.149444 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.150065 13072 raft_consensus.cc:2454] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:27.158021 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:27.158644 13077 raft_consensus.cc:2362] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4. Re-sending same reply.
I20260808 20:14:27.159459 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.160193 15098 raft_consensus.cc:1067] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Signalling peer 78b9263889cd4f1582578a3aa4f644d9 to start an election
I20260808 20:14:27.160488 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:27.161123 14818 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:27.163547 12001 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:27.164934 13078 raft_consensus.cc:2454] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:27.141187 14902 raft_consensus.cc:2726] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:27.167775 13077 raft_consensus.cc:2379] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 6: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:27.145007 12254 raft_consensus.cc:2379] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 3: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:27.170656 12045 raft_consensus.cc:1226] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7. Ops: []
I20260808 20:14:27.142031 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.166277 12000 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:27.173295 12038 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:14:27.169241 13079 raft_consensus.cc:2454] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:27.171106 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.176827 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.177160 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.177663 13073 raft_consensus.cc:2379] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 5: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:27.177775 13079 raft_consensus.cc:2454] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:27.179325 13079 pending_rounds.cc:76] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20: Aborting all ops after (but not including) 3
I20260808 20:14:27.179579 13079 pending_rounds.cc:97] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260808 20:14:27.179832 13079 raft_consensus.cc:2866] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEARNER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 20:14:27.182355 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 245100us
I20260808 20:14:27.173290 11889 catalog_manager.cc:5495] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:27.176213 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.184885 13077 raft_consensus.cc:2379] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 6: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:27.176069 14903 raft_consensus.cc:2726] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:14:27.158043 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.191551 14818 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:27.201555 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 56410.1us
W20260808 20:14:27.202776 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 275495us
I20260808 20:14:27.189267 14902 raft_consensus.cc:2781] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:27.205036 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:27.207440 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:27.225663 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:27.166056 15106 consensus_queue.cc:1046] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260808 20:14:27.193240 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.231197 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:27.200989 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.233999 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:27.189167 15045 consensus_queue.cc:1035] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260808 20:14:27.251905 12043 raft_consensus.cc:2396] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260808 20:14:27.251852 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 67533.2us
W20260808 20:14:27.253939 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 234253us
I20260808 20:14:27.256567 13072 raft_consensus.cc:2379] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 3: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:27.256387 15107 consensus_queue.cc:1035] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.023s
I20260808 20:14:27.218839 13076 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:14:27.258783 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.261204 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.261922 12241 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 1.
I20260808 20:14:27.227382 14902 raft_consensus.cc:686] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:27.263027 14902 consensus_queue.cc:237] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.263259 14891 raft_consensus.cc:504] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.271965 12205 leader_election.cc:302] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:27.272853 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.255548 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:27.282064 12205 leader_election.cc:302] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:27.287416 12206 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:27.286885 13014 leader_election.cc:302] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:27.278707 15106 raft_consensus.cc:3032] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:27.261823 15027 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:27.281397 13073 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEARNER]: Advancing to term 5
I20260808 20:14:27.291743 15027 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:27.293310 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:27.297114 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:27.297631 12248 raft_consensus.cc:2379] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 6: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:27.289491 15106 raft_consensus.cc:729] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:27.299170 15106 consensus_queue.cc:260] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:27.300405 15106 raft_consensus.cc:3037] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:27.301380 12205 leader_election.cc:302] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:27.301522 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.302273 13071 raft_consensus.cc:2454] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:27.303366 14891 leader_election.cc:288] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:27.281174 15117 raft_consensus.cc:2781] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:27.312147 12244 raft_consensus.cc:2379] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:27.313344 12001 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:27.284603 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.266530 15104 raft_consensus.cc:504] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:27.302475 15108 raft_consensus.cc:2781] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:27.320957 13014 leader_election.cc:302] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
W20260808 20:14:27.321341 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 70360.6us
I20260808 20:14:27.321580 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.324391 12041 raft_consensus.cc:2379] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 6: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:27.350019 13015 leader_election.cc:302] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:27.314211 15027 raft_consensus.cc:2781] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260808 20:14:27.350675 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 169244us
I20260808 20:14:27.318778 15117 raft_consensus.cc:482] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:27.324707 15051 raft_consensus.cc:504] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:27.354081 13073 raft_consensus.cc:1261] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEARNER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260808 20:14:27.324795 15116 raft_consensus.cc:2781] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:27.356860 12102 leader_election.cc:302] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:27.345165 15108 raft_consensus.cc:482] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:27.364391 12001 leader_election.cc:302] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:27.367024 15109 raft_consensus.cc:2781] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:27.367483 15109 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:27.367818 15109 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:27.360574 13073 pending_rounds.cc:76] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20: Aborting all ops after (but not including) 3
I20260808 20:14:27.370862 13073 pending_rounds.cc:97] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260808 20:14:27.371119 13073 raft_consensus.cc:2866] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEARNER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:27.372603 12247 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:27.374440 15109 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:27.375886 15109 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:27.377282 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.377779 13076 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:14:27.351958 15118 raft_consensus.cc:2781] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:27.379326 15118 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:27.379617 15118 raft_consensus.cc:3037] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:27.359076 15117 raft_consensus.cc:3037] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:27.356366 15027 raft_consensus.cc:482] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:27.387951 12101 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:27.388810 15104 leader_election.cc:288] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:27.391040 15118 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.392253 15105 raft_consensus.cc:2726] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:27.393817 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.362433 15108 raft_consensus.cc:3037] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:27.349030 14951 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.397083 13015 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:27.361434 15120 raft_consensus.cc:2781] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:27.400614 12102 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:27.401384 15003 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.402997 15051 leader_election.cc:288] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:27.422227 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 57875.7us
W20260808 20:14:27.423774 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 62496.6us
I20260808 20:14:27.424671 12101 leader_election.cc:302] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:27.424271 12001 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:27.426424 12001 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:27.427193 14903 raft_consensus.cc:2781] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:27.427563 14903 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:27.427914 14903 raft_consensus.cc:3037] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:27.424003 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:27.428102 12206 leader_election.cc:302] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:27.428968 13076 raft_consensus.cc:2379] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 3: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:27.429943 15123 raft_consensus.cc:2781] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:27.429080 12038 raft_consensus.cc:1226] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:27.430321 15123 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:27.431200 15123 raft_consensus.cc:3037] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:27.411571 15098 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.022s
I20260808 20:14:27.433359 15051 raft_consensus.cc:2781] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:27.429303 15111 raft_consensus.cc:2781] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:27.434059 15051 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:27.434223 15111 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:27.434490 15051 raft_consensus.cc:3037] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:27.435621 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.435979 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.436318 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:27.422698 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 57067.5us
I20260808 20:14:27.425604 12101 leader_election.cc:302] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:27.399138 15027 raft_consensus.cc:3037] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:27.422745 15120 raft_consensus.cc:482] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:27.441037 15120 raft_consensus.cc:3037] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:27.441377 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:27.442544 15119 log.cc:924] Time spent T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.149s user 0.002s sys 0.000s
I20260808 20:14:27.440213 15111 raft_consensus.cc:3037] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:27.425136 15107 raft_consensus.cc:2781] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:27.445670 15107 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:27.446002 15107 raft_consensus.cc:3037] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:27.450467 12040 raft_consensus.cc:1226] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:27.430294 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.459041 13015 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:27.399034 15116 raft_consensus.cc:686] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:27.405524 14902 raft_consensus.cc:2781] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:27.449005 13073 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
W20260808 20:14:27.465369 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 151010us
I20260808 20:14:27.425468 15003 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:27.422158 15100 raft_consensus.cc:2781] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260808 20:14:27.470247 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 44741.9us
W20260808 20:14:27.460188 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 145376us
I20260808 20:14:27.468241 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.474540 13080 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:27.475757 12205 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:27.468439 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 201183us
I20260808 20:14:27.437744 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.478538 12101 leader_election.cc:302] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:27.461199 15096 raft_consensus.cc:2726] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:27.486451 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.487524 14951 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:27.491873 15027 raft_consensus.cc:504] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.463471 14902 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:27.493131 12251 raft_consensus.cc:1226] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:27.494079 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.494736 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.493405 14902 raft_consensus.cc:3037] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:27.496352 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.496680 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.497273 13078 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
I20260808 20:14:27.497514 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.496352 15027 leader_election.cc:288] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:27.502262 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.503495 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.503989 15111 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:27.505240 15116 consensus_queue.cc:237] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.507015 15120 raft_consensus.cc:504] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.509238 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.509930 13076 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:27.513722 15118 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:27.514652 12101 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
W20260808 20:14:27.516394 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 76099.4us
I20260808 20:14:27.493363 13074 raft_consensus.cc:2379] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
W20260808 20:14:27.520057 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 119509us
I20260808 20:14:27.482916 14777 raft_consensus.cc:2781] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:27.482792 15105 raft_consensus.cc:2726] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:27.477519 15100 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
W20260808 20:14:27.524039 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 48372.3us
I20260808 20:14:27.526117 12102 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:27.526162 12205 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:27.527827 15131 raft_consensus.cc:2781] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:27.528023 15098 raft_consensus.cc:2781] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:27.525445 15027 raft_consensus.cc:2726] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:27.547039 15098 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:27.547435 15098 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:27.541431 15120 leader_election.cc:288] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:27.526628 15100 raft_consensus.cc:3037] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:27.529039 12205 leader_election.cc:302] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:27.517239 15129 raft_consensus.cc:2726] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:27.519696 15117 raft_consensus.cc:504] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.553274 15129 raft_consensus.cc:2781] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:27.553653 15129 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:27.553967 15129 raft_consensus.cc:3037] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:27.523433 14777 raft_consensus.cc:482] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:27.528750 15125 raft_consensus.cc:2781] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:27.535241 15105 consensus_queue.cc:1046] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20260808 20:14:27.538594 14903 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.546288 15131 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:27.500583 15123 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:27.557035 15125 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:27.559352 15125 raft_consensus.cc:3037] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
W20260808 20:14:27.564108 15121 log.cc:924] Time spent T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.261s user 0.002s sys 0.000s
I20260808 20:14:27.558161 15131 raft_consensus.cc:3037] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:14:27.565966 15113 log.cc:924] Time spent T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.276s user 0.000s sys 0.003s
I20260808 20:14:27.570648 14903 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:27.556867 14777 raft_consensus.cc:3037] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:27.573581 15123 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:27.554733 15045 raft_consensus.cc:482] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:27.577173 15125 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:27.577667 15045 raft_consensus.cc:504] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:27.579053 15125 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:27.554500 15117 leader_election.cc:288] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:27.599802 15111 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:27.558676 15027 raft_consensus.cc:2726] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260808 20:14:27.602960 15137 log.cc:924] Time spent T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20260808 20:14:27.581571 15120 raft_consensus.cc:3032] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:27.591209 15051 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.580811 15045 leader_election.cc:288] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:27.555176 15107 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.595722 14777 raft_consensus.cc:504] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.597976 15131 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:27.604055 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 119969us
W20260808 20:14:27.606334 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 48769.7us
I20260808 20:14:27.628250 12205 leader_election.cc:302] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:27.626082 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 100003us
I20260808 20:14:27.632925 12102 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:27.635298 12206 leader_election.cc:302] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:27.636339 15045 raft_consensus.cc:2726] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:27.636658 15104 raft_consensus.cc:2726] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:27.604182 15120 raft_consensus.cc:729] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:27.616263 15051 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:27.614892 15098 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:27.637814 15120 consensus_queue.cc:260] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.558900 14902 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.639722 15098 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:27.640810 14902 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:27.646436 12205 leader_election.cc:302] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:27.647403 15045 consensus_queue.cc:1035] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:14:27.649329 13073 raft_consensus.cc:2379] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:27.650173 15107 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:27.622864 15129 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.655145 15142 raft_consensus.cc:2741] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260808 20:14:27.614483 15100 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:27.656217 15027 raft_consensus.cc:2781] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:27.656597 15027 raft_consensus.cc:482] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:27.656988 15027 raft_consensus.cc:3037] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:27.657291 15100 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:27.603827 15108 raft_consensus.cc:504] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.658249 12206 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:27.659024 15142 consensus_queue.cc:1046] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:14:27.659770 15131 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:27.659991 15104 raft_consensus.cc:3032] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:27.660391 15104 raft_consensus.cc:729] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:27.661242 15104 consensus_queue.cc:260] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:27.661796 14777 leader_election.cc:288] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:27.662595 15104 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:27.664644 15108 leader_election.cc:288] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:27.647548 12102 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:27.664448 15027 raft_consensus.cc:504] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:27.667025 15027 leader_election.cc:288] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:27.667635 14777 raft_consensus.cc:2726] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:27.668942 12102 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:27.669826 14777 raft_consensus.cc:2726] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:27.671604 12102 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:27.672376 14777 raft_consensus.cc:2726] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:27.655061 15129 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:27.674695 12102 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
W20260808 20:14:27.675179 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 156804us
I20260808 20:14:27.681419 12102 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:27.682574 12102 leader_election.cc:302] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
W20260808 20:14:27.684370 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 167278us
W20260808 20:14:27.685045 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 188197us
I20260808 20:14:27.685658 15129 raft_consensus.cc:2781] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:27.688482 12102 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:27.686883 13015 leader_election.cc:302] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:27.687705 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 172302us
I20260808 20:14:27.689867 12102 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:27.691175 12205 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:27.692451 13014 leader_election.cc:398] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Vote denied by peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) with higher term. Message: Invalid argument: T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3.
I20260808 20:14:27.692530 15100 raft_consensus.cc:2781] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:27.694339 15100 raft_consensus.cc:482] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:27.694763 15100 raft_consensus.cc:3037] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:27.696300 15120 raft_consensus.cc:3037] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:27.692770 15105 raft_consensus.cc:2726] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:27.685947 15131 raft_consensus.cc:2726] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:27.693310 15045 raft_consensus.cc:2781] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:27.692803 13014 leader_election.cc:401] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260808 20:14:27.693090 15098 raft_consensus.cc:2781] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:27.698092 15105 raft_consensus.cc:2726] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) with higher term. Message: Invalid argument: T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3.
I20260808 20:14:27.686184 15143 raft_consensus.cc:2781] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:27.676251 14777 consensus_queue.cc:1035] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20260808 20:14:27.697461 15045 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:27.693202 15108 consensus_queue.cc:1046] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:27.698518 15125 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:27.697801 15123 raft_consensus.cc:2726] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:27.699218 15045 raft_consensus.cc:3037] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:27.698246 15098 raft_consensus.cc:686] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:27.698668 15143 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:27.687613 15129 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:27.700474 15108 raft_consensus.cc:3032] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Stepping down as leader of term 6
I20260808 20:14:27.700744 15129 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:27.700914 15108 raft_consensus.cc:729] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:27.700353 15098 consensus_queue.cc:237] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.701812 15108 consensus_queue.cc:260] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:27.699532 15125 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.705750 12105 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:27.699069 13014 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:27.700287 15143 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:27.707432 15108 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:27.709281 15123 raft_consensus.cc:2726] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:27.711184 15125 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:27.712150 12105 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:27.713827 12105 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:27.715615 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:27.715740 12102 leader_election.cc:302] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:27.716482 12040 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:27.719746 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:27.720636 12036 raft_consensus.cc:2379] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 4: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:27.721640 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:27.722926 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:27.723520 12043 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:27.729197 13014 leader_election.cc:302] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:27.733144 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:27.733893 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:27.734506 12041 raft_consensus.cc:3037] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 4
I20260808 20:14:27.735423 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:27.735960 12039 raft_consensus.cc:3037] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:27.738515 12101 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:27.739441 15107 raft_consensus.cc:2781] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:27.739660 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:27.741366 12045 raft_consensus.cc:1226] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:27.742782 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.743932 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.744658 12244 raft_consensus.cc:2379] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 3: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:27.745304 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.745978 12248 raft_consensus.cc:2354] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 4.
I20260808 20:14:27.745898 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.746572 12241 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:27.747941 12040 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 7.
I20260808 20:14:27.739260 12043 raft_consensus.cc:2454] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:14:27.747813 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:27.750275 12038 raft_consensus.cc:3037] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:27.747463 15125 raft_consensus.cc:2781] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:27.756975 15150 raft_consensus.cc:2726] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:27.759625 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.760267 12243 raft_consensus.cc:2454] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:27.761312 15129 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.762611 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.763303 15129 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:27.765606 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.766342 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.766713 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.767452 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:27.767822 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.768162 12241 raft_consensus.cc:2379] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:27.768517 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.769131 12247 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:27.769600 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.761686 15107 raft_consensus.cc:686] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:27.763501 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:27.771126 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 53434.4us
I20260808 20:14:27.771276 12254 raft_consensus.cc:2379] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
W20260808 20:14:27.770864 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 55185.3us
I20260808 20:14:27.772876 12102 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:27.782289 12101 leader_election.cc:302] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:27.797228 12101 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:27.797878 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.758571 15096 raft_consensus.cc:2781] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:27.758643 15116 raft_consensus.cc:2781] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:27.808317 12040 raft_consensus.cc:1226] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:27.809166 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:27.759200 15125 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:27.805023 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.805449 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.758721 14777 raft_consensus.cc:2781] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:27.804293 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:27.814152 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:27.814747 12046 raft_consensus.cc:3037] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:27.815402 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:27.781747 14902 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:27.784072 15129 raft_consensus.cc:2781] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:27.819440 12205 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:27.821599 12205 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:27.811972 12039 raft_consensus.cc:2454] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:27.816471 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:27.818852 12038 raft_consensus.cc:2454] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:27.820596 12244 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:14:27.823665 12244 raft_consensus.cc:595] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:27.823999 12244 raft_consensus.cc:603] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 20:14:27.796571 15111 raft_consensus.cc:2726] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:27.825840 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:27.828099 12047 raft_consensus.cc:2379] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 5: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:27.826699 12040 raft_consensus.cc:1226] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260808 20:14:27.827147 12150 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:14:27.825821 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:27.827385 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:27.804433 15096 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:27.829465 12146 raft_consensus.cc:2379] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:27.807804 15116 raft_consensus.cc:482] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:27.815936 15125 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:27.834406 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.825783 15123 raft_consensus.cc:2781] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:27.828506 14987 raft_consensus.cc:2781] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election won for term 7
I20260808 20:14:27.816162 14777 raft_consensus.cc:482] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:27.825304 15129 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:27.834695 15027 raft_consensus.cc:2781] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:27.827255 14902 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.837404 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:27.824975 15109 consensus_queue.cc:1046] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260808 20:14:27.838627 12105 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:27.835091 15107 consensus_queue.cc:237] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.828742 12241 raft_consensus.cc:1226] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260808 20:14:27.797205 15045 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.844324 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:27.849892 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 51763.1us
W20260808 20:14:27.961786 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 170256us
I20260808 20:14:27.962133 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:27.962519 12105 leader_election.cc:302] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:27.829398 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:27.832536 15096 raft_consensus.cc:3037] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:27.869684 12249 raft_consensus.cc:2354] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 4.
I20260808 20:14:27.835934 15116 raft_consensus.cc:3037] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:27.833705 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:27.851467 14777 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:27.854883 15129 raft_consensus.cc:3037] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:27.829792 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.896850 15100 raft_consensus.cc:504] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:27.798511 15152 consensus_queue.cc:1046] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260808 20:14:27.870175 15146 consensus_queue.cc:1046] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.086s
W20260808 20:14:27.853233 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 83522.4us
I20260808 20:14:27.904536 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:27.942678 12046 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:27.941335 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:27.990959 12152 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:27.847326 15123 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
W20260808 20:14:27.949344 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 151856us
W20260808 20:14:27.950176 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 128637us
I20260808 20:14:27.954087 15125 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:27.955708 12137 raft_consensus.cc:1261] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 20:14:27.957156 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:27.869748 14987 raft_consensus.cc:686] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 7 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:27.963343 12048 raft_consensus.cc:1226] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:27.997049 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:27.997445 15125 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:27.966992 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:27.967859 12041 raft_consensus.cc:2454] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:27.968436 12244 raft_consensus.cc:1226] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260808 20:14:27.998592 12043 raft_consensus.cc:1226] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:27.996860 14987 consensus_queue.cc:237] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:27.998775 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:27.969254 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:27.999155 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:27.969810 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:27.965634 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:27.999961 12147 raft_consensus.cc:2379] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:28.000034 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:28.000639 12243 raft_consensus.cc:2379] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 3: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:28.000836 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:27.971915 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:28.001240 12146 raft_consensus.cc:2454] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
I20260808 20:14:27.981132 15111 raft_consensus.cc:482] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:27.996860 12205 leader_election.cc:302] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:28.000345 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.963765 13014 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:27.963408 15164 raft_consensus.cc:2726] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:28.003711 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:28.004447 12045 raft_consensus.cc:2379] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 4: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:28.004055 15111 raft_consensus.cc:504] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:28.005175 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:28.006013 15111 leader_election.cc:288] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:27.994923 15123 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:27.960872 15143 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:28.006948 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:28.007931 15143 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:28.005575 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:27.949441 15027 raft_consensus.cc:686] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:27.986250 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:27.972705 15150 raft_consensus.cc:3037] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:27.992355 12150 raft_consensus.cc:2379] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:27.972961 15045 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:27.993358 12101 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:27.966135 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:27.965245 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:27.997807 12138 raft_consensus.cc:3037] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:27.969532 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:28.000784 12142 raft_consensus.cc:2379] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 5: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:28.008697 12144 raft_consensus.cc:2379] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:28.040259 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:28.366627 13079 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 1.
W20260808 20:14:28.073527 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 109831us
I20260808 20:14:28.009791 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:28.331998 12000 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:28.367926 13014 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:28.368387 12000 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:28.327394 15027 consensus_queue.cc:237] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:28.347355 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:28.009799 12147 raft_consensus.cc:2362] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4. Re-sending same reply.
I20260808 20:14:27.997956 15125 raft_consensus.cc:482] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:28.319134 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:28.373771 12140 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:28.095409 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:28.373873 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 505264us
I20260808 20:14:28.348563 12046 raft_consensus.cc:1226] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:28.169384 15140 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:28.097337 15100 leader_election.cc:288] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:28.082098 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:28.376991 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:28.038899 14902 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:28.030376 12248 raft_consensus.cc:1261] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260808 20:14:28.350101 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:28.389361 12045 raft_consensus.cc:2379] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 3: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:28.174185 12243 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
W20260808 20:14:28.293684 11994 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.24.3:38391 timed out after 2.205s (SENT) [suppressed 44 similar messages]
I20260808 20:14:28.010238 12205 leader_election.cc:302] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:28.350802 15150 raft_consensus.cc:2726] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:28.009261 12137 raft_consensus.cc:2379] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:28.039150 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:28.351366 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:28.409876 13074 raft_consensus.cc:3037] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 3
I20260808 20:14:28.410526 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:28.411698 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:28.412407 12241 raft_consensus.cc:2379] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 3: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:28.413153 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:28.293182 11889 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:57474 (request call id 333) took 7811 ms. Trace:
W20260808 20:14:28.217219 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 77353us
I20260808 20:14:28.414458 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:28.175268 15098 raft_consensus.cc:3032] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:28.260352 15146 raft_consensus.cc:3032] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Stepping down as leader of term 2
W20260808 20:14:28.357409 15170 log.cc:924] Time spent T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.220s user 0.000s sys 0.000s
I20260808 20:14:28.173723 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:28.335260 15045 raft_consensus.cc:2726] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 20:14:28.319674 15116 raft_consensus.cc:504] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:28.358443 14777 raft_consensus.cc:504] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:28.438088 13016 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:28.229002 11896 catalog_manager.cc:5495] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 1 to 4, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4). New cstate: current_term: 4 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:28.238049 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:28.461702 12146 raft_consensus.cc:2379] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:28.345610 15129 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:28.009649 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:28.065343 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:28.369166 13014 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:28.413812 11889 rpcz_store.cc:276] 0808 20:14:20.481511 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:20.481671 (+ 160us) service_pool.cc:227] Handling call
0808 20:14:28.293116 (+7811445us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 20:14:27.479937 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 20:14:27.480502 (+ 565us) write_op.cc:432] Acquiring schema lock in shared mode
0808 20:14:27.480564 (+ 62us) write_op.cc:435] Acquired schema lock
0808 20:14:27.480582 (+ 18us) tablet.cc:660] Decoding operations
0808 20:14:27.481796 (+ 1214us) write_op.cc:620] Acquiring the partition lock for write op
0808 20:14:27.481909 (+ 113us) write_op.cc:641] Partition lock acquired for write op
0808 20:14:27.481935 (+ 26us) tablet.cc:683] Acquiring locks for 11 operations
0808 20:14:27.553118 (+ 71183us) tablet.cc:699] Row locks acquired
0808 20:14:27.553157 (+ 39us) write_op.cc:260] PREPARE: finished
0808 20:14:27.553315 (+ 158us) write_op.cc:270] Start()
0808 20:14:27.553461 (+ 146us) write_op.cc:276] Timestamp: P: 1786220067553253 usec, L: 0
0808 20:14:27.553480 (+ 19us) op_driver.cc:331] REPLICATION: starting
0808 20:14:27.554158 (+ 678us) log.cc:842] Serialized 4439 byte log entry
0808 20:14:27.796084 (+241926us) op_driver.cc:444] REPLICATION: finished
0808 20:14:27.952381 (+156297us) write_op.cc:301] APPLY: starting
0808 20:14:27.952481 (+ 100us) tablet.cc:1365] starting BulkCheckPresence
0808 20:14:27.952724 (+ 243us) tablet.cc:1368] finished BulkCheckPresence
0808 20:14:27.952830 (+ 106us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 20:14:27.954160 (+ 1330us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 20:14:27.954385 (+ 225us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0808 20:14:27.954440 (+ 55us) write_op.cc:312] APPLY: finished
0808 20:14:27.954717 (+ 277us) log.cc:842] Serialized 83 byte log entry
0808 20:14:27.964818 (+ 10101us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:14:27.983167 (+ 18349us) write_op.cc:454] Released schema lock
0808 20:14:27.985274 (+ 2107us) write_op.cc:341] FINISH: Updating metrics
0808 20:14:28.009458 (+ 24184us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:14:28.023250 (+ 13792us) write_op.cc:454] Released schema lock
0808 20:14:28.241669 (+218419us) spinlock_profiling.cc:243] Waited 34.3 ms on lock 0x7b5400002dd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d7689 00007fa968275804 00007fa9682757b3 00007fa968275728 00007fa9682756e3 00007fa9682749cc 00007fa96829ea16 00007fa9682865cb 00007fa968270c8d 00007fa968274665
Metrics: {"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":142067,"apply.run_cpu_time_us":3740,"apply.run_wall_time_us":96151,"num_ops":11,"prepare.queue_time_us":76048,"prepare.run_cpu_time_us":19513,"prepare.run_wall_time_us":272911,"raft.queue_time_us":1286,"raft.run_cpu_time_us":12880,"raft.run_wall_time_us":536167,"replication_time_us":233349,"spinlock_wait_cycles":78931200,"thread_start_us":387716,"threads_started":4,"wal-append.queue_time_us":165369}]]}
I20260808 20:14:28.370954 15163 raft_consensus.cc:2726] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:28.371143 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:28.372066 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:28.470156 12251 raft_consensus.cc:1261] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:14:28.465746 11896 catalog_manager.cc:5495] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 4 to 5, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 5 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } }
I20260808 20:14:28.372539 15051 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:28.026298 12249 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:28.475714 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:28.476438 12249 raft_consensus.cc:2354] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 4.
I20260808 20:14:27.994340 12036 raft_consensus.cc:2379] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:28.022511 15166 raft_consensus.cc:2781] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:28.382750 12046 raft_consensus.cc:2379] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 2: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:28.420271 15098 raft_consensus.cc:729] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:28.037137 15161 raft_consensus.cc:3032] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Stepping down as leader of term 4
W20260808 20:14:28.394608 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 100956us
I20260808 20:14:28.002454 12247 raft_consensus.cc:1226] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20260808 20:14:28.328655 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 335437us
I20260808 20:14:28.399262 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:28.499823 13074 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:28.399451 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:28.399893 15111 consensus_queue.cc:1046] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:28.400635 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:28.401309 15142 raft_consensus.cc:2781] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:28.400430 12207 leader_election.cc:302] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
W20260808 20:14:28.400830 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 390653us
W20260808 20:14:28.504045 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 103691us
I20260808 20:14:28.405953 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:28.504851 12207 leader_election.cc:302] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:28.409173 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:28.418774 12140 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:28.508508 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:28.420867 12244 raft_consensus.cc:2379] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 3: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:28.357941 15123 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:28.436643 15164 consensus_queue.cc:1046] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260808 20:14:28.463212 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:28.394184 15140 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:28.467852 12152 raft_consensus.cc:2379] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 2: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:28.096720 15120 consensus_queue.cc:1046] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.337s
I20260808 20:14:28.360275 15096 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:28.434739 15150 consensus_queue.cc:1046] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20260808 20:14:28.479975 11896 catalog_manager.cc:5495] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 2 to 3, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4). New cstate: current_term: 3 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:28.389871 12248 pending_rounds.cc:76] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a: Aborting all ops after (but not including) 3
I20260808 20:14:28.484625 15166 raft_consensus.cc:686] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:28.483194 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:28.420531 15146 raft_consensus.cc:729] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:28.420509 14902 raft_consensus.cc:2781] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:28.546994 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:28.388950 15125 raft_consensus.cc:504] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:28.463227 15105 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:28.461702 14777 leader_election.cc:288] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:28.487468 15161 raft_consensus.cc:729] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:28.495224 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:28.496279 13071 raft_consensus.cc:1226] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:28.497952 15129 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:28.485340 15163 raft_consensus.cc:2726] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:28.500577 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:28.400501 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:28.507107 12105 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.010s sys 0.008s
W20260808 20:14:28.572661 12105 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.599s user 0.010s sys 0.010s
I20260808 20:14:28.514317 12140 raft_consensus.cc:2379] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 5: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:28.517448 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:28.573649 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:28.574247 12140 raft_consensus.cc:3037] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
W20260808 20:14:28.529315 11994 leader_election.cc:334] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Timed out: RequestConsensusVote RPC to 127.10.24.3:38391 timed out after 2.173s (SENT)
I20260808 20:14:28.529922 12207 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:28.094924 15165 raft_consensus.cc:2726] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 20:14:28.532136 12205 leader_election.cc:302] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:28.527786 14987 raft_consensus.cc:2726] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:28.547304 14902 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:28.536811 12248 pending_rounds.cc:97] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260808 20:14:28.538393 12045 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:28.538013 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:28.543551 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 175233us
W20260808 20:14:28.546248 12039 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s user 0.044s sys 0.026s
I20260808 20:14:28.606546 12152 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 1.
W20260808 20:14:28.606544 12039 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.633s user 0.044s sys 0.027s
I20260808 20:14:28.608355 12000 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:28.506340 15132 raft_consensus.cc:2726] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:28.492453 12101 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:28.547652 12245 raft_consensus.cc:2454] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:28.606199 12248 raft_consensus.cc:2866] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:28.547547 11896 catalog_manager.cc:5495] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 5 to 6, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 6 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:14:28.548224 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:28.610031 12146 raft_consensus.cc:3037] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
W20260808 20:14:28.556584 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 118583us
I20260808 20:14:28.610988 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:28.509554 15142 raft_consensus.cc:686] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:28.528000 15108 raft_consensus.cc:2781] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:28.537146 12141 raft_consensus.cc:2379] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:28.567816 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:28.561558 12147 raft_consensus.cc:2354] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 4. Current term is 5.
I20260808 20:14:28.528257 12243 raft_consensus.cc:2362] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5. Re-sending same reply.
I20260808 20:14:28.568207 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:28.533730 15117 consensus_queue.cc:1046] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260808 20:14:28.592324 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:28.594013 15096 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:28.534778 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:28.604228 14987 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:28.618995 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 89707.4us
I20260808 20:14:28.605046 12105 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:28.607973 14902 raft_consensus.cc:3037] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
W20260808 20:14:28.557276 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 43754.3us
I20260808 20:14:28.558810 15105 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:28.630818 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
W20260808 20:14:28.631464 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 262397us
I20260808 20:14:28.614768 11896 catalog_manager.cc:5495] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 4 to 5, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 5 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } }
I20260808 20:14:28.596218 15150 raft_consensus.cc:482] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:28.623131 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 93281.1us
I20260808 20:14:28.653787 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
W20260808 20:14:28.615357 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 122984us
I20260808 20:14:28.654637 12142 raft_consensus.cc:2379] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:28.654795 12101 leader_election.cc:302] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:28.621336 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
W20260808 20:14:28.627377 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 188725us
I20260808 20:14:28.616128 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
W20260808 20:14:28.629631 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 97578.8us
I20260808 20:14:28.630540 12105 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:28.624186 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:28.645097 11896 catalog_manager.cc:5495] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 1 to 2, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 2 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:28.614571 15104 raft_consensus.cc:2781] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:28.625010 12137 raft_consensus.cc:2354] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3.
I20260808 20:14:28.621089 12251 raft_consensus.cc:2454] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:28.619923 15129 raft_consensus.cc:2781] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:28.644714 13016 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:28.661947 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:28.663105 12105 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:28.663808 15096 raft_consensus.cc:2726] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 20:14:28.662444 12101 leader_election.cc:302] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:28.655876 15178 raft_consensus.cc:2726] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:28.705868 15116 leader_election.cc:288] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:28.696712 12140 raft_consensus.cc:2454] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:28.669375 12138 raft_consensus.cc:2454] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
W20260808 20:14:28.798060 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 55413.8us
I20260808 20:14:28.721660 12039 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:54212
I20260808 20:14:28.844576 12039 raft_consensus.cc:595] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:28.844878 12039 raft_consensus.cc:603] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 20:14:28.706467 11896 catalog_manager.cc:5495] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 4 to 5, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 5 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:28.629536 14987 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:28.617918 15185 raft_consensus.cc:2781] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:28.803663 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:28.643668 15161 consensus_queue.cc:260] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:28.705472 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:28.712061 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:28.852716 12244 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:28.723431 15123 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:28.820845 12144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1568) took 1013 ms. Trace:
W20260808 20:14:28.714239 15171 log.cc:924] Time spent T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.147s user 0.000s sys 0.000s
I20260808 20:14:28.668694 15143 raft_consensus.cc:3037] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:28.723896 15142 consensus_queue.cc:237] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:28.703095 12207 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:28.723078 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:28.721158 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:28.721910 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:28.717646 15105 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:28.567085 15098 consensus_queue.cc:260] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:28.722249 12043 raft_consensus.cc:1261] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 20:14:28.704766 12041 raft_consensus.cc:2379] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:28.703761 12046 raft_consensus.cc:1226] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:28.725163 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:28.849671 15185 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:28.830833 12142 raft_consensus.cc:2454] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 7.
W20260808 20:14:28.833173 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 220665us
I20260808 20:14:28.835773 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:28.851142 15027 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:28.834697 15192 consensus_queue.cc:1035] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
W20260808 20:14:28.839150 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 176779us
I20260808 20:14:28.845671 15140 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:28.644701 15146 consensus_queue.cc:260] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:28.844161 13014 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:28.848349 12045 raft_consensus.cc:2454] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:28.876353 12045 raft_consensus.cc:1226] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:28.711236 15108 raft_consensus.cc:686] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:28.667670 15166 consensus_queue.cc:237] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:28.845947 15174 log.cc:924] Time spent T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.286s user 0.000s sys 0.001s
I20260808 20:14:28.801898 15150 raft_consensus.cc:504] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:28.870776 15185 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:28.889616 12254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1683) took 1094 ms. Trace:
I20260808 20:14:28.890077 12254 rpcz_store.cc:276] 0808 20:14:27.795574 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:27.795731 (+ 157us) service_pool.cc:227] Handling call
0808 20:14:27.815720 (+ 19989us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:27.826286 (+ 10566us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:27.826312 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:27.826701 (+ 389us) log.cc:842] Serialized 26 byte log entry
0808 20:14:28.072214 (+245513us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:28.072262 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:28.072343 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:28.889394 (+817051us) raft_consensus.cc:1671] finished
0808 20:14:28.889414 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:28.889533 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":223383,"threads_started":1,"wal-append.queue_time_us":649792}
I20260808 20:14:28.832448 15117 raft_consensus.cc:3032] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:28.856753 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:28.877709 15168 log.cc:924] Time spent T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.273s user 0.000s sys 0.001s
I20260808 20:14:28.833107 15191 raft_consensus.cc:2781] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:28.857103 11896 catalog_manager.cc:5495] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 1 to 3, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 3 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } }
I20260808 20:14:28.862212 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:28.862641 12245 raft_consensus.cc:2379] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:28.705163 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:28.905112 12152 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:28.866645 12046 raft_consensus.cc:1261] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 20:14:28.853305 15125 leader_election.cc:288] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:28.912467 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:28.913693 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:28.868408 12140 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:28.914885 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
W20260808 20:14:28.835917 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 191299us
I20260808 20:14:28.868605 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:28.868948 12000 leader_election.cc:302] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:28.871572 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
W20260808 20:14:28.873427 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 170419us
I20260808 20:14:28.860869 15123 raft_consensus.cc:2781] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:28.858772 15198 raft_consensus.cc:482] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:28.856837 15129 raft_consensus.cc:686] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:28.860736 15096 raft_consensus.cc:2726] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:28.877257 12102 leader_election.cc:302] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:28.875020 13075 raft_consensus.cc:2379] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:28.879401 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:28.883328 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:28.884322 12251 raft_consensus.cc:1226] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:28.885041 12243 raft_consensus.cc:1226] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:28.887279 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:28.847512 15120 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:28.856276 12144 rpcz_store.cc:276] 0808 20:14:27.803338 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:27.805615 (+ 2277us) service_pool.cc:227] Handling call
0808 20:14:28.803392 (+997777us) spinlock_profiling.cc:243] Waited 25.9 ms on lock 0x7b54003ed354. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:28.816640 (+ 13248us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":59470720}
I20260808 20:14:28.904302 12047 raft_consensus.cc:2379] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 3: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:28.916159 15108 consensus_queue.cc:237] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:28.874536 15027 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:29.049297 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:29.214597 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:29.322078 12152 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:29.051313 11896 catalog_manager.cc:5495] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 3 to 4, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 4 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:14:29.259478 12245 raft_consensus.cc:1261] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 20:14:28.917804 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:28.921952 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:28.893538 15051 raft_consensus.cc:2726] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:28.889639 15150 leader_election.cc:288] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:29.274906 13016 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:29.179200 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 381655us
I20260808 20:14:28.917450 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:29.334393 12152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1595) took 1013 ms. Trace:
W20260808 20:14:29.159660 12146 consensus_meta.cc:287] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.549s user 0.007s sys 0.002s
W20260808 20:14:29.074128 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 193684us
I20260808 20:14:28.921653 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:29.048641 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 172073us
I20260808 20:14:29.339902 12146 raft_consensus.cc:2454] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
W20260808 20:14:29.326742 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 449450us
I20260808 20:14:29.334326 12150 raft_consensus.cc:3037] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:29.338063 12142 raft_consensus.cc:2354] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3.
I20260808 20:14:29.335176 11896 catalog_manager.cc:5495] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 4 to 7, leader changed from e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 7 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:29.331059 13077 raft_consensus.cc:3037] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:28.949506 12140 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
I20260808 20:14:29.341867 12140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1592) took 1021 ms. Trace:
I20260808 20:14:29.342307 12140 rpcz_store.cc:276] 0808 20:14:28.320137 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:28.835539 (+515402us) service_pool.cc:227] Handling call
0808 20:14:29.341809 (+506270us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:29.343297 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:29.343900 12040 raft_consensus.cc:1226] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 4. Ops: []
W20260808 20:14:29.050969 15180 log.cc:924] Time spent T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.348s user 0.001s sys 0.000s
I20260808 20:14:28.896286 12048 raft_consensus.cc:1226] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:29.351815 12140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 263) took 1024 ms. Trace:
I20260808 20:14:28.891083 15105 consensus_queue.cc:1046] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.450s
I20260808 20:14:29.326530 15123 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
W20260808 20:14:29.186023 14902 consensus_meta.cc:287] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.530s user 0.008s sys 0.000s
I20260808 20:14:29.217602 15045 raft_consensus.cc:3032] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
W20260808 20:14:29.347671 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 684613us
I20260808 20:14:29.166530 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:29.216032 13071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53296 (request call id 644) took 1239 ms. Trace:
I20260808 20:14:29.351028 15107 raft_consensus.cc:3032] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:29.348361 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 262) took 1020 ms. Trace:
I20260808 20:14:29.142620 15104 raft_consensus.cc:686] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
W20260808 20:14:29.349628 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 480729us
I20260808 20:14:29.339727 12152 rpcz_store.cc:276] 0808 20:14:28.321067 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:28.661544 (+340477us) service_pool.cc:227] Handling call
0808 20:14:29.334338 (+672794us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:28.918064 15197 raft_consensus.cc:2726] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:29.250649 15098 raft_consensus.cc:3037] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:29.340584 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1593) took 1020 ms. Trace:
I20260808 20:14:29.356078 11896 catalog_manager.cc:5495] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 3 to 4, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 4 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } }
I20260808 20:14:29.344164 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:29.285461 14987 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:29.352314 12043 raft_consensus.cc:1226] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:28.895776 15117 raft_consensus.cc:729] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:29.352257 12140 rpcz_store.cc:276] 0808 20:14:28.327797 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.343048 (+1015251us) service_pool.cc:227] Handling call
0808 20:14:29.351762 (+ 8714us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:29.216202 15145 raft_consensus.cc:2781] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:29.283512 15161 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:29.213344 12247 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1691) took 1253 ms. Trace:
I20260808 20:14:29.216885 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:29.361994 15146 raft_consensus.cc:3037] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:29.157876 15191 raft_consensus.cc:686] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:29.431563 15191 consensus_queue.cc:237] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:29.363405 15107 raft_consensus.cc:729] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:29.376310 15100 raft_consensus.cc:3032] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:29.360507 12047 raft_consensus.cc:1226] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260808 20:14:29.361526 12105 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:29.363476 12142 rpcz_store.cc:276] 0808 20:14:28.327572 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:28.917571 (+589999us) service_pool.cc:227] Handling call
0808 20:14:29.348293 (+430722us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:29.358839 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260808 20:14:29.371564 12146 rpcz_store.cc:276] 0808 20:14:28.320455 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:28.531728 (+211273us) service_pool.cc:227] Handling call
0808 20:14:29.340527 (+808799us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:29.372880 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:29.374387 12041 raft_consensus.cc:1226] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:29.374655 12000 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:29.375850 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:29.374588 12045 raft_consensus.cc:1226] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:29.377846 12254 raft_consensus.cc:1261] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 20:14:29.440611 12146 raft_consensus.cc:1226] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:29.441588 12043 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:29.442009 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:14:29.442801 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:14:29.443358 15210 raft_consensus.cc:2726] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:14:29.376765 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:29.246433 15140 consensus_queue.cc:1035] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.118s
I20260808 20:14:29.445740 13079 raft_consensus.cc:2454] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 1.
W20260808 20:14:29.442906 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 68340.3us
W20260808 20:14:29.388522 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 113735us
I20260808 20:14:29.361405 12144 raft_consensus.cc:1226] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:29.448087 12144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1586) took 1479 ms. Trace:
I20260808 20:14:29.333940 12207 leader_election.cc:302] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:29.360669 15123 raft_consensus.cc:3037] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:29.448401 12144 rpcz_store.cc:276] 0808 20:14:27.968296 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.356517 (+1388221us) service_pool.cc:227] Handling call
0808 20:14:29.361231 (+ 4714us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:29.447843 (+ 86612us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:29.448035 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:29.451016 13078 raft_consensus.cc:2379] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:29.441973 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1598) took 1116 ms. Trace:
I20260808 20:14:29.355165 15051 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:29.380357 12137 raft_consensus.cc:2379] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:29.358994 15203 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:29.451678 12146 rpcz_store.cc:276] 0808 20:14:28.325173 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.440127 (+1114954us) service_pool.cc:227] Handling call
0808 20:14:29.440420 (+ 293us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:29.441709 (+ 1289us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:29.441913 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:29.452124 15203 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:29.359120 15204 raft_consensus.cc:2781] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:29.452860 12146 raft_consensus.cc:1226] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:29.453521 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1600) took 1127 ms. Trace:
I20260808 20:14:29.453819 12146 rpcz_store.cc:276] 0808 20:14:28.325701 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.452425 (+1126724us) service_pool.cc:227] Handling call
0808 20:14:29.452709 (+ 284us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:29.453286 (+ 577us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:29.453471 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:29.454259 15203 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:29.360915 15045 raft_consensus.cc:729] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:29.455080 12146 raft_consensus.cc:1261] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 20:14:29.455610 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1603) took 1129 ms. Trace:
I20260808 20:14:29.455374 15045 consensus_queue.cc:260] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:29.455928 12146 rpcz_store.cc:276] 0808 20:14:28.326541 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.454497 (+1127956us) service_pool.cc:227] Handling call
0808 20:14:29.454771 (+ 274us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:29.455401 (+ 630us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:29.455569 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:29.456848 15045 raft_consensus.cc:3037] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:29.359941 15219 consensus_queue.cc:1035] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.066s
I20260808 20:14:29.425683 15230 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:29.355815 15212 raft_consensus.cc:2781] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:29.363065 15163 consensus_queue.cc:1046] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.066s
I20260808 20:14:29.428571 15117 consensus_queue.cc:260] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:29.361840 13072 raft_consensus.cc:2454] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:29.359699 15027 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:29.360921 14902 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:29.360093 15227 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:29.337116 15198 raft_consensus.cc:504] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:29.482141 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:29.375804 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:29.376538 13075 raft_consensus.cc:1261] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260808 20:14:29.437418 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 87799.1us
I20260808 20:14:29.448655 12000 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:29.459124 15125 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
W20260808 20:14:29.459705 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 98248.8us
W20260808 20:14:29.459363 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 125512us
I20260808 20:14:29.460480 13077 raft_consensus.cc:2454] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:29.462597 15228 consensus_queue.cc:1046] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260808 20:14:29.470580 15143 consensus_meta.cc:287] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.612s user 0.001s sys 0.007s
I20260808 20:14:29.464164 15109 raft_consensus.cc:2726] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:29.377295 15132 consensus_queue.cc:1035] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.087s
I20260808 20:14:29.461966 15164 raft_consensus.cc:2781] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election won for term 3
W20260808 20:14:29.469977 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 120586us
I20260808 20:14:29.473839 15117 raft_consensus.cc:3037] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:29.473951 15163 raft_consensus.cc:3032] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:29.477586 15186 consensus_queue.cc:1046] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260808 20:14:29.477790 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:29.404978 15145 raft_consensus.cc:686] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:29.479977 12142 raft_consensus.cc:1226] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260808 20:14:29.528069 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1591) took 1235 ms. Trace:
I20260808 20:14:29.480305 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 417) took 1051 ms. Trace:
I20260808 20:14:29.361650 13071 rpcz_store.cc:276] 0808 20:14:27.976133 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:27.981299 (+ 5166us) service_pool.cc:227] Handling call
0808 20:14:28.008992 (+ 27693us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:28.579420 (+570428us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:29.215707 (+636287us) spinlock_profiling.cc:243] Waited 476 ms on lock 0x7b540044cad4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:29.215935 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1095342336}
I20260808 20:14:29.528414 12142 rpcz_store.cc:276] 0808 20:14:28.292488 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.438993 (+1146505us) service_pool.cc:227] Handling call
0808 20:14:29.470966 (+ 31973us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:29.527787 (+ 56821us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:29.528012 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:29.484719 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 135462us
I20260808 20:14:29.204847 12141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1577) took 1239 ms. Trace:
I20260808 20:14:29.484947 12205 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:29.437827 15100 raft_consensus.cc:729] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:29.424274 15202 raft_consensus.cc:2781] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260808 20:14:29.493402 11994 leader_election.cc:334] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Timed out: RequestConsensusVote RPC to 127.10.24.3:38391 timed out after 1.731s (SENT)
I20260808 20:14:29.497323 13016 leader_election.cc:398] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139) with higher term. Message: Invalid argument: T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3.
I20260808 20:14:29.532621 13016 leader_election.cc:401] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260808 20:14:29.503996 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 55405.7us
I20260808 20:14:29.533403 12000 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:29.533595 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:29.534099 15145 consensus_queue.cc:237] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:29.534968 12000 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:29.535966 12000 leader_election.cc:302] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:29.471288 14777 raft_consensus.cc:3032] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:29.465345 15212 raft_consensus.cc:482] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:29.469122 15051 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:29.522826 15140 raft_consensus.cc:2781] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:29.435999 15116 consensus_queue.cc:1035] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.056s
I20260808 20:14:29.525640 15228 raft_consensus.cc:482] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:29.525471 12244 raft_consensus.cc:3037] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:29.528571 12207 leader_election.cc:302] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:29.529064 12040 rpcz_store.cc:276] 0808 20:14:28.417244 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:28.417422 (+ 178us) service_pool.cc:227] Handling call
0808 20:14:28.417709 (+ 287us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:29.348192 (+930483us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:29.468226 (+120034us) spinlock_profiling.cc:243] Waited 926 ms on lock 0x7b54004844d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:29.468384 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2129699712}
I20260808 20:14:29.498255 14987 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:29.496713 15204 raft_consensus.cc:686] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:29.511358 12147 raft_consensus.cc:2454] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:29.532411 11994 leader_election.cc:302] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:29.513438 15143 raft_consensus.cc:2726] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:29.547794 12142 raft_consensus.cc:1226] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:29.629086 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 271) took 1299 ms. Trace:
I20260808 20:14:29.434964 15104 consensus_queue.cc:237] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:29.548538 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 63697.1us
I20260808 20:14:29.631605 12140 raft_consensus.cc:1261] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:14:29.552506 12001 leader_election.cc:302] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:29.547199 15210 raft_consensus.cc:2726] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:29.556210 15192 raft_consensus.cc:2726] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:29.545936 14777 raft_consensus.cc:729] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:29.560142 15237 consensus_queue.cc:1046] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260808 20:14:29.484163 12137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 276) took 1150 ms. Trace:
I20260808 20:14:29.619161 15228 raft_consensus.cc:504] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:29.546283 12138 raft_consensus.cc:2379] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:29.531883 12036 raft_consensus.cc:1226] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:29.617002 12147 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 260) took 1290 ms. Trace:
I20260808 20:14:29.729362 12147 rpcz_store.cc:276] 0808 20:14:28.327002 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.214250 (+887248us) service_pool.cc:227] Handling call
0808 20:14:29.570106 (+355856us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:29.729815 15219 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:29.730302 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:29.537027 15164 raft_consensus.cc:686] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:29.730446 15219 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:29.731668 15164 consensus_queue.cc:237] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:29.732776 15219 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:29.735432 15210 raft_consensus.cc:2726] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:29.525372 15163 raft_consensus.cc:729] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:29.620714 15187 raft_consensus.cc:2726] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139) with higher term. Message: Invalid argument: T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3.
I20260808 20:14:29.736106 15210 raft_consensus.cc:2781] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:29.532306 15202 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:29.513077 15125 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:29.623229 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 65966.6us
W20260808 20:14:29.623768 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 124996us
W20260808 20:14:29.627702 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 99194.6us
I20260808 20:14:29.738227 15228 leader_election.cc:288] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:29.738690 12207 leader_election.cc:302] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:29.739410 15238 consensus_queue.cc:1046] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 20:14:29.740118 14777 consensus_queue.cc:260] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:29.741096 15202 raft_consensus.cc:3037] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:29.741366 14777 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:29.469182 15129 consensus_queue.cc:237] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:29.468537 15230 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:29.629423 12142 rpcz_store.cc:276] 0808 20:14:28.329675 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.530566 (+1200891us) service_pool.cc:227] Handling call
0808 20:14:29.547636 (+ 17070us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:29.628823 (+ 81187us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:29.629025 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:29.745043 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:29.400209 12247 rpcz_store.cc:276] 0808 20:14:27.959465 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:27.961591 (+ 2126us) service_pool.cc:227] Handling call
0808 20:14:27.981308 (+ 19717us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:28.500875 (+519567us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:29.213073 (+712198us) spinlock_profiling.cc:243] Waited 9.29 ms on lock 0x7b54002ebb50. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d9a09 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:29.213262 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21403648}
I20260808 20:14:29.547361 15212 raft_consensus.cc:3037] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:29.618855 14987 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:29.531672 12141 rpcz_store.cc:276] 0808 20:14:27.965755 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:27.965880 (+ 125us) service_pool.cc:227] Handling call
0808 20:14:29.204543 (+1238663us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b540030cd54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:29.204748 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":319081728}
W20260808 20:14:29.633410 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 104720us
W20260808 20:14:29.622642 15185 consensus_meta.cc:287] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.719s user 0.008s sys 0.001s
I20260808 20:14:29.645560 15140 raft_consensus.cc:686] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:29.556774 15111 consensus_queue.cc:1046] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.074s
W20260808 20:14:29.559311 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 62122.9us
I20260808 20:14:29.788734 13016 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:29.493377 15227 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:29.651974 12244 raft_consensus.cc:2454] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:29.790803 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:29.655098 15192 raft_consensus.cc:2781] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:14:29.655956 15236 raft_consensus.cc:2781] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:29.656432 15206 raft_consensus.cc:2726] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:29.657372 15237 raft_consensus.cc:3032] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:29.736510 15210 raft_consensus.cc:482] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:29.808303 13016 leader_election.cc:398] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Vote denied by peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139) with higher term. Message: Invalid argument: T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 4. Current term is 5.
I20260808 20:14:29.757395 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:29.567240 15116 consensus_queue.cc:1225] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer 4cd46f13bceb4c3c9ef71e755e17b995 log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260808 20:14:29.765811 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 229887us
I20260808 20:14:29.783479 12254 raft_consensus.cc:1226] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260808 20:14:29.786434 12142 raft_consensus.cc:1226] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:29.787509 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:29.627072 15123 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:29.797444 12250 raft_consensus.cc:2379] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:29.753862 12138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 259) took 1420 ms. Trace:
I20260808 20:14:29.650573 12137 rpcz_store.cc:276] 0808 20:14:28.330979 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:28.857734 (+526755us) service_pool.cc:227] Handling call
0808 20:14:29.481039 (+623305us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:29.821178 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:29.821074 15140 consensus_queue.cc:237] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:29.821627 15227 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:29.822019 12137 raft_consensus.cc:2379] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:29.822927 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:29.823966 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:29.824119 15193 raft_consensus.cc:2726] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260808 20:14:29.825999 15241 log.cc:924] Time spent T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260808 20:14:29.779765 15109 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:29.758054 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:29.812405 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 272) took 1482 ms. Trace:
I20260808 20:14:29.812743 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:29.811966 12000 leader_election.cc:302] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:29.817724 13016 leader_election.cc:398] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139) with higher term. Message: Invalid argument: T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3.
I20260808 20:14:29.818822 12101 leader_election.cc:302] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:29.793293 15051 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:29.800501 15236 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:29.830776 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:29.831033 12142 rpcz_store.cc:276] 0808 20:14:28.329904 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.746006 (+1416102us) service_pool.cc:227] Handling call
0808 20:14:29.786283 (+ 40277us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:29.812212 (+ 25929us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:29.812360 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:29.835028 13077 raft_consensus.cc:2454] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:29.836927 12207 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:29.840372 12249 raft_consensus.cc:2379] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
W20260808 20:14:29.841291 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 288872us
I20260808 20:14:29.805917 15237 raft_consensus.cc:729] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:29.843078 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:29.843645 13077 raft_consensus.cc:2362] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2. Re-sending same reply.
I20260808 20:14:29.844259 12101 leader_election.cc:302] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:29.845958 12038 raft_consensus.cc:1261] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260808 20:14:29.846585 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:29.849066 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:29.849831 12038 raft_consensus.cc:2379] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 5: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:29.808794 12048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1339) took 1149 ms. Trace:
I20260808 20:14:29.812521 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
W20260808 20:14:29.862044 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 44273.7us
I20260808 20:14:29.804852 15198 leader_election.cc:288] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:29.862027 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:29.865307 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:29.865715 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:29.811568 15210 raft_consensus.cc:3037] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:29.809844 15192 raft_consensus.cc:686] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:29.818439 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:29.951560 12150 consensus_meta.cc:287] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.610s user 0.011s sys 0.003s
W20260808 20:14:29.951969 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 458548us
I20260808 20:14:29.841105 15145 raft_consensus.cc:2781] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:14:29.860484 15045 raft_consensus.cc:2726] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260808 20:14:29.895180 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 47271.5us
I20260808 20:14:29.894266 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:29.895180 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 53106.8us
I20260808 20:14:29.792203 15185 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:30.078675 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:30.081925 12251 raft_consensus.cc:3037] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 3
I20260808 20:14:29.961140 13077 raft_consensus.cc:2379] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:29.951920 12150 raft_consensus.cc:2454] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:30.083161 12150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 275) took 1752 ms. Trace:
I20260808 20:14:30.083494 12150 rpcz_store.cc:276] 0808 20:14:28.330715 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:28.863642 (+532927us) service_pool.cc:227] Handling call
0808 20:14:30.083097 (+1219455us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:30.084388 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:30.004269 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:30.078444 15246 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:30.085664 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:30.086037 15246 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:30.086254 15264 consensus_queue.cc:1035] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.133s
I20260808 20:14:30.086717 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:30.087744 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:30.088236 15246 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:29.836014 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:30.054281 15051 raft_consensus.cc:2781] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:30.090168 15051 raft_consensus.cc:686] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:29.861013 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:30.091328 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:30.089774 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:30.092308 12137 raft_consensus.cc:2354] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5.
I20260808 20:14:29.830085 15243 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:30.059655 15146 consensus_meta.cc:287] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.622s user 0.010s sys 0.003s
W20260808 20:14:29.894830 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 49427.4us
I20260808 20:14:30.061555 12001 leader_election.cc:302] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:30.054656 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 72141.8us
I20260808 20:14:30.059746 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:30.069301 12142 raft_consensus.cc:2379] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:29.950889 15164 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:29.840163 15236 raft_consensus.cc:3037] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:29.865978 15228 raft_consensus.cc:2726] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:29.861704 15193 raft_consensus.cc:2781] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:29.865736 15187 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:29.814580 15132 consensus_queue.cc:1035] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.132s
I20260808 20:14:30.091006 15051 consensus_queue.cc:237] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:30.055501 11994 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:30.157886 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 102488us
I20260808 20:14:29.898485 15191 raft_consensus.cc:482] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:30.095613 12150 raft_consensus.cc:2354] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5.
I20260808 20:14:29.989234 15145 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:30.093369 12000 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:29.840226 15109 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:30.054497 12101 leader_election.cc:302] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:30.167505 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 155739us
I20260808 20:14:30.096639 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:29.808636 15204 consensus_queue.cc:237] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:30.100124 12001 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:30.102065 12039 raft_consensus.cc:1226] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:29.819576 12138 rpcz_store.cc:276] 0808 20:14:28.326766 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.375550 (+1048784us) service_pool.cc:227] Handling call
0808 20:14:29.746798 (+371248us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:30.059660 15096 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:30.174435 12150 raft_consensus.cc:1226] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:30.175079 12150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 274) took 1844 ms. Trace:
I20260808 20:14:30.175372 12150 rpcz_store.cc:276] 0808 20:14:28.330448 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:30.161516 (+1831068us) service_pool.cc:227] Handling call
0808 20:14:30.174258 (+ 12742us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:30.174861 (+ 603us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:30.175026 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:29.866429 12048 rpcz_store.cc:276] 0808 20:14:28.659056 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:28.661765 (+ 2709us) service_pool.cc:227] Handling call
0808 20:14:28.877805 (+216040us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:29.364750 (+486945us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:29.790281 (+425531us) spinlock_profiling.cc:243] Waited 73.3 ms on lock 0x7b54003b4f50. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d9a09 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:29.804731 (+ 14450us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":168695168}
I20260808 20:14:30.103637 15193 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:30.136353 13016 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:30.081698 15140 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:30.101414 15247 raft_consensus.cc:3032] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:29.828580 15163 consensus_queue.cc:260] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:30.069736 13071 raft_consensus.cc:2362] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3. Re-sending same reply.
I20260808 20:14:30.088251 12249 raft_consensus.cc:3037] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:30.200789 15247 raft_consensus.cc:729] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:30.201846 15247 consensus_queue.cc:260] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:30.203056 15247 raft_consensus.cc:3037] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:30.168764 12101 leader_election.cc:302] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:30.103694 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:30.205412 15111 raft_consensus.cc:2726] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:30.206224 12101 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:30.206760 12137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 278) took 1875 ms. Trace:
I20260808 20:14:30.207028 15111 raft_consensus.cc:2726] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:14:30.208529 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:30.098378 12141 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:30.171859 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:30.214438 12249 raft_consensus.cc:2396] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260808 20:14:30.207401 12101 leader_election.cc:302] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:30.157974 14902 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:30.217202 12101 leader_election.cc:302] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:30.226052 12150 raft_consensus.cc:1226] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:30.226622 12150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 277) took 1895 ms. Trace:
I20260808 20:14:30.161883 15264 raft_consensus.cc:2726] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:30.226924 12150 rpcz_store.cc:276] 0808 20:14:28.331241 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:30.176092 (+1844851us) service_pool.cc:227] Handling call
0808 20:14:30.225886 (+ 49794us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:30.226403 (+ 517us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:30.226574 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:30.215415 15111 raft_consensus.cc:2726] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:30.167321 15145 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:30.185232 15246 raft_consensus.cc:2726] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:30.238912 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:30.092857 15243 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:30.216425 15193 raft_consensus.cc:3037] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:30.266363 12150 raft_consensus.cc:2379] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:30.188095 15140 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:30.270403 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 111244us
I20260808 20:14:30.230221 15108 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:30.229526 15228 raft_consensus.cc:2726] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:30.105093 15164 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:30.225237 14987 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:30.207118 12137 rpcz_store.cc:276] 0808 20:14:28.331502 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:30.205934 (+1874432us) service_pool.cc:227] Handling call
0808 20:14:30.206250 (+ 316us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:30.206438 (+ 188us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:30.206467 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:30.206508 (+ 41us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:30.206548 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:30.206603 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:30.206708 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:30.277756 12141 raft_consensus.cc:1226] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:30.275935 11994 leader_election.cc:302] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:30.158171 15187 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:30.280429 12251 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:30.236397 15125 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:30.283887 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 183827us
I20260808 20:14:30.317619 12251 raft_consensus.cc:1226] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:30.216624 15096 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:30.185810 15191 raft_consensus.cc:504] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:30.278149 15240 log.cc:924] Time spent T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.589s user 0.000s sys 0.001s
I20260808 20:14:30.335134 12152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1587) took 2366 ms. Trace:
W20260808 20:14:30.335265 14777 consensus_meta.cc:287] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.593s user 0.007s sys 0.000s
I20260808 20:14:30.335481 12152 rpcz_store.cc:276] 0808 20:14:27.968563 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.364874 (+1396311us) service_pool.cc:227] Handling call
0808 20:14:29.365209 (+ 335us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:29.386940 (+ 21731us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:29.387643 (+ 703us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:29.424873 (+ 37230us) log.cc:842] Serialized 26 byte log entry
0808 20:14:29.462333 (+ 37460us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:29.462369 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:29.462430 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:30.334938 (+872508us) raft_consensus.cc:1671] finished
0808 20:14:30.334957 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:30.335078 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":37092,"threads_started":1,"wal-append.queue_time_us":108034}
I20260808 20:14:30.280153 15263 raft_consensus.cc:2726] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 20:14:30.329886 15243 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:30.337528 15120 consensus_meta.cc:287] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.998s user 0.008s sys 0.001s
I20260808 20:14:30.337996 15120 raft_consensus.cc:2726] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:30.338634 13073 raft_consensus.cc:1226] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:30.269264 15264 raft_consensus.cc:2781] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 20:14:30.340416 15264 raft_consensus.cc:686] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:30.341266 15264 consensus_queue.cc:237] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:30.342131 15107 consensus_queue.cc:260] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:30.345985 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:30.346639 13075 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
W20260808 20:14:30.347020 15242 log.cc:924] Time spent T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.593s user 0.000s sys 0.002s
I20260808 20:14:30.347543 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:30.348184 13075 raft_consensus.cc:2454] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:30.349076 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:30.350138 12140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1588) took 2381 ms. Trace:
I20260808 20:14:30.350515 12140 rpcz_store.cc:276] 0808 20:14:27.968826 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.377403 (+1408577us) service_pool.cc:227] Handling call
0808 20:14:29.378113 (+ 710us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:29.631962 (+253849us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:30.349887 (+717925us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x7b540018f454. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:30.350078 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":582257408}
I20260808 20:14:30.355487 12001 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:30.356318 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:30.357223 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:30.315671 11994 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:30.102216 15237 consensus_queue.cc:260] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:30.272320 12138 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:53162
I20260808 20:14:30.319450 15269 consensus_queue.cc:1046] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.189s
W20260808 20:14:30.324656 15265 log.cc:924] Time spent T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.232s user 0.001s sys 0.000s
I20260808 20:14:30.314560 15230 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:30.337476 15263 raft_consensus.cc:2726] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:30.223068 15192 consensus_queue.cc:237] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:30.325596 15278 raft_consensus.cc:2781] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:14:30.327189 15164 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:30.327996 15267 log.cc:924] Time spent T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
W20260808 20:14:30.328794 15261 log.cc:924] Time spent T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.224s user 0.000s sys 0.000s
W20260808 20:14:30.329439 15250 log.cc:924] Time spent T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.347s user 0.001s sys 0.000s
I20260808 20:14:30.329842 15109 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:30.281124 15108 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:30.330681 12048 raft_consensus.cc:1261] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260808 20:14:30.331737 15117 consensus_meta.cc:287] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.806s user 0.009s sys 0.000s
W20260808 20:14:30.332340 15161 consensus_meta.cc:287] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.932s user 0.007s sys 0.000s
I20260808 20:14:30.333529 15142 consensus_queue.cc:1035] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 20:14:30.339900 15140 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:30.235255 14902 consensus_queue.cc:1046] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
W20260808 20:14:30.362309 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 226034us
I20260808 20:14:30.424480 13016 leader_election.cc:302] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:30.424669 13074 raft_consensus.cc:2454] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:30.364400 12251 raft_consensus.cc:1261] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 20:14:30.368001 12043 raft_consensus.cc:2379] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:30.315923 15146 raft_consensus.cc:2726] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 20:14:30.430123 15258 raft_consensus.cc:1050] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
W20260808 20:14:30.430622 15258 raft_consensus.cc:1054] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995: Unable to promote non-voter e5bd9a4616ce4acfa20dc7a002984a20: Illegal state: Leader has not yet committed an operation in its own term
I20260808 20:14:30.431120 13080 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:30.371996 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:30.375385 15185 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:30.369257 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 53768.8us
I20260808 20:14:30.432328 14902 raft_consensus.cc:3032] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:30.433871 11994 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:30.433918 14902 raft_consensus.cc:729] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:30.434895 14902 consensus_queue.cc:260] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:30.435595 11994 leader_election.cc:302] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:30.436067 14902 raft_consensus.cc:3037] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:30.394272 15238 consensus_queue.cc:1046] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260808 20:14:30.436858 11994 leader_election.cc:398] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Vote denied by peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) with higher term. Message: Invalid argument: T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 4.
I20260808 20:14:30.439035 11994 leader_election.cc:401] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260808 20:14:30.439131 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1347) took 1563 ms. Trace:
I20260808 20:14:30.439280 15187 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:30.441099 15145 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:30.442091 13077 raft_consensus.cc:1226] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 7. Current term is 8. Ops: []
I20260808 20:14:30.442705 15108 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:30.437062 15185 raft_consensus.cc:2726] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 20:14:30.444974 15290 raft_consensus.cc:2781] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:30.445461 15290 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:30.445816 15290 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:30.378921 15123 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:30.451511 12047 raft_consensus.cc:1261] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260808 20:14:30.446883 12039 rpcz_store.cc:276] 0808 20:14:28.875324 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:28.896114 (+ 20790us) service_pool.cc:227] Handling call
0808 20:14:28.896393 (+ 279us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:30.184971 (+1288578us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:30.438891 (+253920us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b5400170cd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:30.439046 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2772417280}
I20260808 20:14:30.437870 15258 raft_consensus.cc:3037] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:30.452978 15096 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:30.388651 15278 raft_consensus.cc:686] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:30.455186 15096 consensus_queue.cc:1035] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:14:30.456538 15263 raft_consensus.cc:2726] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) with higher term. Message: Invalid argument: T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 4.
I20260808 20:14:30.456175 15278 consensus_queue.cc:237] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:30.457619 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:30.458207 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 270) took 2128 ms. Trace:
I20260808 20:14:30.458309 15161 raft_consensus.cc:2781] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:30.461057 15161 raft_consensus.cc:686] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:30.461920 15161 consensus_queue.cc:237] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:30.465646 12101 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:30.458335 12039 raft_consensus.cc:2379] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 7: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:30.458547 12146 rpcz_store.cc:276] 0808 20:14:28.329431 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.456629 (+1127198us) service_pool.cc:227] Handling call
0808 20:14:29.457078 (+ 449us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:29.457308 (+ 230us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:29.457333 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:29.457732 (+ 399us) log.cc:842] Serialized 26 byte log entry
0808 20:14:29.783967 (+326235us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:29.784003 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:29.784072 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:30.458020 (+673948us) raft_consensus.cc:1671] finished
0808 20:14:30.458039 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:30.458153 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":325834,"threads_started":1,"wal-append.queue_time_us":372730}
W20260808 20:14:30.472803 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 48410.7us
I20260808 20:14:30.474936 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:30.381103 15230 raft_consensus.cc:2726] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:30.484027 11994 leader_election.cc:302] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:30.438745 15202 consensus_meta.cc:287] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.697s user 0.008s sys 0.000s
I20260808 20:14:30.494973 12140 raft_consensus.cc:1226] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:30.502110 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:30.505858 12048 raft_consensus.cc:1261] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260808 20:14:30.507822 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 414507us
I20260808 20:14:30.508560 12000 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:30.482683 12144 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.025s sys 0.017s
I20260808 20:14:30.512990 11994 leader_election.cc:302] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:30.507081 12248 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.677s user 0.000s sys 0.011s
W20260808 20:14:30.520494 12248 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.690s user 0.000s sys 0.012s
W20260808 20:14:30.549463 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 51692.6us
I20260808 20:14:30.520475 15247 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:30.553077 15247 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:30.554593 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 273) took 2224 ms. Trace:
I20260808 20:14:30.555069 15247 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:30.554975 12142 rpcz_store.cc:276] 0808 20:14:28.330159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:30.094203 (+1764044us) service_pool.cc:227] Handling call
0808 20:14:30.135147 (+ 40944us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 20:14:30.135498 (+ 351us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:30.135524 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 20:14:30.136038 (+ 514us) log.cc:842] Serialized 52 byte log entry
0808 20:14:30.359183 (+223145us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:30.359221 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:30.359283 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:30.554392 (+195109us) raft_consensus.cc:1671] finished
0808 20:14:30.554410 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:30.554537 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":21260,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":3989,"thread_start_us":136461,"threads_started":1,"wal-append.queue_time_us":347353}
I20260808 20:14:30.486490 15163 raft_consensus.cc:3037] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:30.561620 12152 raft_consensus.cc:1261] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260808 20:14:30.562422 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 63612.3us
I20260808 20:14:30.549094 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:30.563025 12146 raft_consensus.cc:2454] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:30.558903 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:30.564301 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:30.560074 12040 raft_consensus.cc:1261] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260808 20:14:30.566215 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:30.566856 12146 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:30.560179 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:30.567514 12251 raft_consensus.cc:2454] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:30.568264 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:30.480578 15285 consensus_queue.cc:1046] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.151s
W20260808 20:14:30.589437 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 80964.7us
I20260808 20:14:30.593535 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
W20260808 20:14:30.549714 12144 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.603s user 0.025s sys 0.018s
I20260808 20:14:30.699431 15287 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:30.496886 15123 raft_consensus.cc:2726] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:14:30.496377 15185 consensus_queue.cc:1035] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.204s
I20260808 20:14:30.700057 15287 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:30.627641 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:30.628849 12000 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:30.629149 12254 raft_consensus.cc:3037] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
W20260808 20:14:30.744748 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 181351us
I20260808 20:14:30.745471 12038 pending_rounds.cc:76] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting all ops after (but not including) 3
I20260808 20:14:30.745633 13016 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:30.745821 12038 pending_rounds.cc:97] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260808 20:14:30.746927 12038 raft_consensus.cc:2866] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:30.747687 12045 raft_consensus.cc:1226] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 1. Current term is 2. Ops: []
W20260808 20:14:30.749751 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 121002us
I20260808 20:14:30.750385 12000 leader_election.cc:302] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:30.556877 12138 raft_consensus.cc:465] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Not starting forced leader election -- already a leader
I20260808 20:14:30.563719 15198 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:30.485142 15187 consensus_queue.cc:1035] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.103s
I20260808 20:14:30.627194 15289 raft_consensus.cc:3037] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:30.497097 15152 consensus_queue.cc:1046] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.261s
I20260808 20:14:30.761397 12040 raft_consensus.cc:1261] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 20:14:30.762714 13016 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:30.704408 15230 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:14:30.765533 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 252666us
I20260808 20:14:30.765180 12207 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:30.768694 12001 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:30.706491 15161 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:30.628129 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:30.777973 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:30.757179 15140 raft_consensus.cc:2781] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:30.757754 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:30.820299 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:30.810189 11994 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:30.811790 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 54605.8us
I20260808 20:14:30.628254 15243 consensus_queue.cc:1046] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.147s
I20260808 20:14:30.758755 15264 raft_consensus.cc:2781] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:30.764372 15191 leader_election.cc:288] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:30.521270 15202 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:30.816920 12206 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.007s sys 0.012s
W20260808 20:14:30.816036 15212 consensus_meta.cc:287] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 1.025s user 0.001s sys 0.008s
W20260808 20:14:30.818331 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 55709.6us
I20260808 20:14:30.770221 15051 raft_consensus.cc:2781] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election won for term 3
W20260808 20:14:30.822211 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 57111.5us
I20260808 20:14:30.822813 12101 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:30.773768 15269 raft_consensus.cc:2781] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260808 20:14:30.828277 12206 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.555s user 0.008s sys 0.012s
W20260808 20:14:30.844446 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 572102us
I20260808 20:14:30.795120 15278 consensus_queue.cc:1046] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.031s
I20260808 20:14:30.793057 15185 consensus_queue.cc:1046] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20260808 20:14:30.843822 15125 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:30.846443 12101 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:30.887087 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 43624.3us
I20260808 20:14:30.819063 12243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1794) took 1063 ms. Trace:
I20260808 20:14:30.854923 15269 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:30.892524 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:30.864476 13078 raft_consensus.cc:1226] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 1. Current term is 2. Ops: []
W20260808 20:14:30.903510 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 134882us
I20260808 20:14:30.863943 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:30.818395 15140 raft_consensus.cc:482] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:30.825214 15228 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:30.884856 15298 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:30.828502 15145 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:30.806650 15263 consensus_queue.cc:1046] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260808 20:14:30.893329 15269 raft_consensus.cc:3037] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:30.763447 15198 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:30.854656 15243 consensus_queue.cc:1225] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer b9d94f35ea15421aa1d721201c782d8a log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260808 20:14:30.855031 15096 raft_consensus.cc:2726] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:30.841077 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1340) took 2179 ms. Trace:
I20260808 20:14:30.819759 15161 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:30.890322 12101 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
W20260808 20:14:30.864830 12206 leader_election.cc:334] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.539s (SENT)
I20260808 20:14:30.961572 12206 leader_election.cc:302] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:30.872628 15276 log.cc:924] Time spent T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.275s user 0.000s sys 0.001s
W20260808 20:14:30.911268 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 101162us
I20260808 20:14:30.855165 15117 consensus_queue.cc:1046] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.038s
I20260808 20:14:30.960712 12036 rpcz_store.cc:276] 0808 20:14:28.659371 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:28.665749 (+ 6378us) service_pool.cc:227] Handling call
0808 20:14:28.798262 (+132513us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:29.729335 (+931073us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:30.820346 (+1091011us) spinlock_profiling.cc:243] Waited 709 ms on lock 0x7b54004646d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:30.838575 (+ 18229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1631608192}
I20260808 20:14:30.974526 15255 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:30.975054 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:30.974985 15255 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:30.840162 15212 raft_consensus.cc:504] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:30.891705 15125 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:30.977406 15314 log.cc:924] Time spent T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.165s user 0.001s sys 0.000s
I20260808 20:14:30.978299 15212 leader_election.cc:288] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:30.889650 15098 consensus_meta.cc:287] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 1.504s user 0.011s sys 0.002s
I20260808 20:14:30.895327 13079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 416) took 1105 ms. Trace:
I20260808 20:14:30.889101 15249 raft_consensus.cc:482] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:30.828708 15285 raft_consensus.cc:2781] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:30.934157 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:30.980422 12250 raft_consensus.cc:2454] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:30.885025 15283 raft_consensus.cc:1050] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:14:30.939589 15334 consensus_queue.cc:1035] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:30.894023 15164 raft_consensus.cc:2726] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:30.842157 15292 raft_consensus.cc:2781] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:30.996971 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:30.941866 14902 raft_consensus.cc:2726] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 20:14:30.925395 15051 raft_consensus.cc:686] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:31.009017 12040 raft_consensus.cc:1226] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:31.009300 15051 consensus_queue.cc:237] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:31.010344 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:14:30.933626 15045 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:30.974089 15332 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:30.979125 15320 log.cc:924] Time spent T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.118s user 0.000s sys 0.000s
I20260808 20:14:30.925630 15140 raft_consensus.cc:3037] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:30.979046 12152 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:30.991293 15255 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:30.985491 13015 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:30.978901 15316 log.cc:924] Time spent T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.138s user 0.000s sys 0.000s
I20260808 20:14:30.979034 13079 rpcz_store.cc:276] 0808 20:14:29.789867 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.805472 (+ 15605us) service_pool.cc:227] Handling call
0808 20:14:30.344293 (+538821us) spinlock_profiling.cc:243] Waited 5.65 ms on lock 0x7b5800005198. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:30.344476 (+ 183us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:30.344652 (+ 176us) raft_consensus.cc:1520] Early marking committed up to index 4
0808 20:14:30.558311 (+213659us) log.cc:842] Serialized 14 byte log entry
0808 20:14:30.895031 (+336720us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:30.895079 (+ 48us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:30.895120 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:30.895176 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:30.895277 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19936,"prepare.queue_time_us":384924,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":154589,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":108,"spinlock_wait_cycles":37556736,"thread_start_us":522901,"threads_started":3,"wal-append.queue_time_us":381408}
I20260808 20:14:30.998440 15096 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:31.002381 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 112163us
I20260808 20:14:30.950016 15142 consensus_queue.cc:1035] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20260808 20:14:30.982100 14777 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:31.006685 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 43975us
I20260808 20:14:30.947896 15228 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:31.007347 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 208408us
I20260808 20:14:31.037563 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:31.038322 12142 raft_consensus.cc:2379] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 2: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:31.008183 15322 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:30.956230 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:31.012884 12045 pending_rounds.cc:76] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting all ops after (but not including) 4
I20260808 20:14:30.983237 15243 raft_consensus.cc:2781] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:30.978911 15125 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:31.049292 11994 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:30.948696 15264 raft_consensus.cc:686] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:30.951697 15191 consensus_queue.cc:1046] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260808 20:14:30.999464 12036 raft_consensus.cc:1226] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:30.979607 15249 raft_consensus.cc:504] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:31.034580 15230 raft_consensus.cc:2726] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260808 20:14:31.017927 12244 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53030 (request call id 333) took 2131 ms (client timeout 2177 ms). Trace:
I20260808 20:14:31.013424 15117 raft_consensus.cc:2726] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 20:14:31.030907 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 166128us
I20260808 20:14:31.017323 15164 consensus_queue.cc:1046] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260808 20:14:31.032712 15202 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:14:31.033934 12101 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:31.072451 15321 log.cc:924] Time spent T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.079s user 0.000s sys 0.001s
I20260808 20:14:31.034042 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:31.041558 12045 pending_rounds.cc:97] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20260808 20:14:31.073819 12045 raft_consensus.cc:2866] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:30.956063 15305 consensus_queue.cc:1046] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.229s
I20260808 20:14:31.024920 12247 raft_consensus.cc:1226] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 2. Current term is 3. Ops: []
W20260808 20:14:31.044463 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 59064.7us
I20260808 20:14:31.075232 15202 raft_consensus.cc:3032] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:31.075775 15202 raft_consensus.cc:729] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
W20260808 20:14:31.082198 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 50132.4us
I20260808 20:14:31.079723 12152 raft_consensus.cc:2454] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 8.
I20260808 20:14:31.049911 15243 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:31.051221 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
W20260808 20:14:31.051731 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 100453us
I20260808 20:14:31.087095 12140 raft_consensus.cc:2379] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 2: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:31.088171 12140 raft_consensus.cc:1261] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260808 20:14:30.951274 15298 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:31.055511 11994 leader_election.cc:302] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:31.057052 15212 raft_consensus.cc:2726] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:14:30.988698 15292 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:31.061952 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:30.889568 12243 rpcz_store.cc:276] 0808 20:14:29.753620 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.779552 (+ 25932us) service_pool.cc:227] Handling call
0808 20:14:29.834934 (+ 55382us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:29.947740 (+112806us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:29.954717 (+ 6977us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:30.166546 (+211829us) log.cc:842] Serialized 26 byte log entry
0808 20:14:30.390751 (+224205us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:30.392796 (+ 2045us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:30.436962 (+ 44166us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:30.796783 (+359821us) raft_consensus.cc:1671] finished
0808 20:14:30.796825 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:30.816693 (+ 19868us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":141153,"threads_started":1,"wal-append.queue_time_us":344948}
I20260808 20:14:31.018500 15045 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:31.036365 15198 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:31.035068 15096 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:31.074615 15337 raft_consensus.cc:2781] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:30.958297 15306 consensus_queue.cc:1035] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.233s
W20260808 20:14:31.077154 12206 leader_election.cc:334] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.687s (SENT)
I20260808 20:14:31.051875 14902 consensus_queue.cc:1035] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20260808 20:14:31.077266 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:31.082917 12101 leader_election.cc:302] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:31.039557 13076 raft_consensus.cc:2379] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 3: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:31.086179 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:31.090534 12000 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:31.062422 12244 rpcz_store.cc:269] 0808 20:14:28.882513 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:28.883102 (+ 589us) service_pool.cc:227] Handling call
0808 20:14:31.014072 (+2130970us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b54001b0b54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:31.014176 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":442741120}
I20260808 20:14:31.042339 14777 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:31.063905 15322 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:31.078145 12254 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:31.021555 15332 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:31.308279 11994 leader_election.cc:302] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:31.263055 12046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:53014 (request call id 1401) took 2543 ms. Trace:
I20260808 20:14:31.329893 12249 raft_consensus.cc:1226] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260808 20:14:31.331542 12254 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1807) took 1574 ms (client timeout 1786 ms). Trace:
W20260808 20:14:31.328022 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 245163us
I20260808 20:14:31.231467 12250 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1804) took 1475 ms. Trace:
I20260808 20:14:31.333389 12250 rpcz_store.cc:276] 0808 20:14:29.756204 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:30.003934 (+247730us) service_pool.cc:227] Handling call
0808 20:14:31.231265 (+1227331us) spinlock_profiling.cc:243] Waited 886 ms on lock 0x7b540007e254. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:31.231374 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2038905600}
I20260808 20:14:31.334695 12251 raft_consensus.cc:1261] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260808 20:14:31.257418 12147 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1295) took 1604 ms (client timeout 1983 ms). Trace:
W20260808 20:14:31.258050 13080 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53290 (request call id 497) took 2365 ms (client timeout 2024 ms). Trace:
I20260808 20:14:31.084831 15204 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
W20260808 20:14:31.303290 15210 consensus_meta.cc:287] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 1.353s user 0.011s sys 0.000s
W20260808 20:14:31.326470 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 236012us
I20260808 20:14:31.326895 15230 raft_consensus.cc:2726] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:31.092000 15164 consensus_queue.cc:1225] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer 4cd46f13bceb4c3c9ef71e755e17b995 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260808 20:14:31.328049 11994 leader_election.cc:302] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:31.296038 15298 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:31.331895 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:31.445513 12040 raft_consensus.cc:2379] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 4: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:31.328263 12046 rpcz_store.cc:276] 0808 20:14:28.719183 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.002203 (+283020us) service_pool.cc:227] Handling call
0808 20:14:31.262479 (+2260276us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400170cd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825faf0 00007fa9682e2f2a 00007fa96d21fc6e 00007fa966916807 00007fa966916792 00007fa9669166c1 00007fa96691663c 00007fa9669158f2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:31.262968 (+ 489us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2705994624}
I20260808 20:14:31.368721 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:31.447094 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:31.101711 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:31.335480 13080 rpcz_store.cc:269] 0808 20:14:28.893037 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:28.893174 (+ 137us) service_pool.cc:227] Handling call
0808 20:14:31.257812 (+2364638us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400321c54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:31.257914 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2498438016}
I20260808 20:14:31.325892 12146 raft_consensus.cc:3037] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:31.434128 12150 raft_consensus.cc:2396] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 20:14:31.090019 15243 raft_consensus.cc:3037] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
W20260808 20:14:31.289033 15295 log.cc:924] Time spent T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20260808 20:14:31.328794 15129 raft_consensus.cc:3037] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:31.434234 13079 raft_consensus.cc:1226] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 1. Current term is 2. Ops: []
W20260808 20:14:31.221127 15313 log.cc:924] Time spent T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.686s user 0.005s sys 0.000s
W20260808 20:14:31.172392 15327 log.cc:924] Time spent T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.266s user 0.001s sys 0.000s
W20260808 20:14:31.290702 15258 consensus_meta.cc:287] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.838s user 0.007s sys 0.004s
I20260808 20:14:31.447851 12244 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
W20260808 20:14:31.461378 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 394397us
I20260808 20:14:31.097460 15292 raft_consensus.cc:3037] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:31.219655 15347 raft_consensus.cc:2726] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:31.442927 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:31.317389 15228 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:31.194370 15315 log.cc:924] Time spent T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.246s user 0.000s sys 0.001s
I20260808 20:14:31.223631 15289 raft_consensus.cc:2726] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:31.319554 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1354) took 1835 ms. Trace:
I20260808 20:14:31.461992 12047 raft_consensus.cc:2379] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 2: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
W20260808 20:14:31.333071 12254 rpcz_store.cc:269] 0808 20:14:29.764067 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.840497 (+ 76430us) service_pool.cc:227] Handling call
0808 20:14:30.593377 (+752880us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b58001a1398. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:31.331485 (+738108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":53973888}
I20260808 20:14:31.232757 13072 raft_consensus.cc:1261] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 20:14:31.303937 15300 raft_consensus.cc:482] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:31.174537 15212 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:31.262248 15246 raft_consensus.cc:3032] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:31.502789 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:31.237891 15290 consensus_meta.cc:287] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.792s user 0.007s sys 0.004s
I20260808 20:14:31.101058 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:31.236768 15045 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:31.238511 15236 consensus_meta.cc:287] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 1.081s user 0.007s sys 0.006s
I20260808 20:14:31.173074 15161 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:31.468782 12146 raft_consensus.cc:2454] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:31.329085 15185 raft_consensus.cc:2726] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260808 20:14:31.335192 12147 rpcz_store.cc:269] 0808 20:14:29.652790 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.730049 (+ 77259us) service_pool.cc:227] Handling call
0808 20:14:31.257232 (+1527183us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b540022f1d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:31.257362 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":311081856}
I20260808 20:14:31.333015 15337 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:31.472229 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:31.254457 15339 raft_consensus.cc:2781] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:31.477766 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:31.257762 15249 leader_election.cc:288] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:31.482400 12206 leader_election.cc:334] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.745s (SENT)
I20260808 20:14:31.471704 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
W20260808 20:14:31.182641 12241 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1706) took 2788 ms (client timeout 1911 ms). Trace:
I20260808 20:14:31.136693 15198 consensus_queue.cc:1035] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.331s
I20260808 20:14:31.486703 13015 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:31.488099 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:31.221987 15349 consensus_queue.cc:1046] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.396s
I20260808 20:14:31.490764 15271 raft_consensus.cc:3032] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:31.487026 15228 consensus_queue.cc:1035] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260808 20:14:31.494738 15096 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:31.193320 15352 consensus_queue.cc:1035] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.426s
I20260808 20:14:31.496130 13071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53290 (request call id 512) took 1394 ms. Trace:
I20260808 20:14:31.496621 14777 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:31.497393 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:31.151844 15142 consensus_queue.cc:1035] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.342s
I20260808 20:14:31.482484 13078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 414) took 1698 ms. Trace:
I20260808 20:14:31.463922 15230 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:31.574226 15140 consensus_meta.cc:287] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.543s user 0.003s sys 0.008s
I20260808 20:14:31.574611 15230 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:31.574770 15140 raft_consensus.cc:504] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:31.467988 14987 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:31.577178 12048 raft_consensus.cc:1226] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:31.577340 14987 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:31.471264 15258 raft_consensus.cc:2726] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 20:14:31.485361 15193 consensus_meta.cc:287] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 1.205s user 0.009s sys 0.003s
I20260808 20:14:31.182108 15358 consensus_queue.cc:1035] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.332s
I20260808 20:14:31.506251 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:31.585971 12047 raft_consensus.cc:2454] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:14:31.514995 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:31.518074 12244 raft_consensus.cc:2362] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5. Re-sending same reply.
I20260808 20:14:31.517081 13072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 413) took 1733 ms. Trace:
W20260808 20:14:31.513933 15163 consensus_meta.cc:287] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.880s user 0.009s sys 0.002s
I20260808 20:14:31.504648 15212 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:31.504047 14902 raft_consensus.cc:3037] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:31.546373 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:31.546464 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:31.607856 12147 raft_consensus.cc:2454] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:14:31.317502 15202 consensus_queue.cc:260] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:31.483559 12041 rpcz_store.cc:276] 0808 20:14:29.483865 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.489044 (+ 5179us) service_pool.cc:227] Handling call
0808 20:14:29.545673 (+ 56629us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:29.818370 (+272697us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:29.827218 (+ 8848us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:29.827665 (+ 447us) log.cc:842] Serialized 26 byte log entry
0808 20:14:30.156127 (+328462us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:30.164256 (+ 8129us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:31.318930 (+1154674us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x7b5400012cd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:31.319020 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:31.319087 (+ 67us) raft_consensus.cc:1671] finished
0808 20:14:31.319129 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:31.319460 (+ 331us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1701,"raft.queue_time_us":13988,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":130,"spinlock_wait_cycles":562320256,"thread_start_us":269829,"threads_started":1,"wal-append.queue_time_us":406492,"wal-append.run_cpu_time_us":1751,"wal-append.run_wall_time_us":1128064}
I20260808 20:14:31.552981 15339 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:31.517540 15337 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:31.506598 15290 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:31.513247 12137 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:31.503134 15246 raft_consensus.cc:729] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:31.435148 15210 raft_consensus.cc:504] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:31.506188 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:31.556130 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 82084.4us
W20260808 20:14:31.615868 12206 leader_election.cc:334] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.895s (SENT)
I20260808 20:14:31.170701 15264 consensus_queue.cc:237] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:31.517464 15236 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:31.557449 15243 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:31.558756 15332 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:31.627200 12245 raft_consensus.cc:2354] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6.
I20260808 20:14:31.497311 15289 consensus_queue.cc:1035] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 20:14:31.316143 15269 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:31.560740 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 74133.8us
W20260808 20:14:31.461951 13076 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57330 (request call id 417) took 1630 ms (client timeout 2055 ms). Trace:
I20260808 20:14:31.563021 15161 raft_consensus.cc:2726] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:31.564721 15322 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:31.635524 15243 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:31.567122 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:31.468940 15283 consensus_queue.cc:237] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 3, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:31.513206 15300 raft_consensus.cc:504] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:31.519207 12241 rpcz_store.cc:269] 0808 20:14:28.393919 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:28.412944 (+ 19025us) service_pool.cc:227] Handling call
0808 20:14:31.182263 (+2769319us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b5400142b54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:31.182481 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3204706560}
I20260808 20:14:31.644505 15246 consensus_queue.cc:260] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:31.646025 15290 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:31.646328 15246 raft_consensus.cc:3037] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:31.627148 12243 raft_consensus.cc:2454] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:31.572968 15271 raft_consensus.cc:729] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:31.570261 12249 raft_consensus.cc:3032] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:31.573972 13078 rpcz_store.cc:276] 0808 20:14:29.784073 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.784491 (+ 418us) service_pool.cc:227] Handling call
0808 20:14:29.784771 (+ 280us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:30.904082 (+1119311us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:31.482276 (+578194us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54005b4854. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:31.482431 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2462492416}
I20260808 20:14:31.580263 12040 raft_consensus.cc:2379] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 2: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:31.591632 12102 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:31.596084 12244 raft_consensus.cc:1226] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:31.611958 15339 raft_consensus.cc:3037] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:31.612211 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:31.607723 12046 raft_consensus.cc:2454] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:31.618822 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:31.589237 15142 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:31.621193 12206 leader_election.cc:334] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 2.005s (SENT)
I20260808 20:14:31.630465 12039 raft_consensus.cc:1226] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:31.651268 12249 raft_consensus.cc:729] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:31.597893 15281 raft_consensus.cc:482] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:31.673065 12206 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:31.632601 12041 raft_consensus.cc:1226] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:31.633064 13073 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53290 (request call id 500) took 2735 ms. Trace:
I20260808 20:14:31.633929 15269 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:14:31.630453 13076 rpcz_store.cc:269] 0808 20:14:29.807634 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.807770 (+ 136us) service_pool.cc:227] Handling call
0808 20:14:30.996792 (+1189022us) spinlock_profiling.cc:243] Waited 1.7 ms on lock 0x7b5800005198. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:31.436148 (+439356us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3911936}
W20260808 20:14:31.643299 15326 log.cc:924] Time spent T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.768s user 0.001s sys 0.011s
I20260808 20:14:31.696487 13076 raft_consensus.cc:1261] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 20:14:31.639746 15230 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:31.595468 15193 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:31.644675 15300 leader_election.cc:288] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:31.686108 15329 log.cc:924] Time spent T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.697s user 0.005s sys 0.000s
W20260808 20:14:31.686597 15323 log.cc:924] Time spent T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.743s user 0.003s sys 0.003s
I20260808 20:14:31.690639 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:31.692163 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:31.731734 12147 raft_consensus.cc:2379] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 5: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:31.732549 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:31.693216 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:31.571911 13071 rpcz_store.cc:276] 0808 20:14:30.101200 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:30.209500 (+108300us) service_pool.cc:227] Handling call
0808 20:14:30.209828 (+ 328us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:30.382076 (+172248us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:30.382110 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:30.444112 (+ 62002us) log.cc:842] Serialized 26 byte log entry
0808 20:14:30.510510 (+ 66398us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:30.510548 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:30.510633 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:31.195432 (+684799us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:31.495659 (+300227us) raft_consensus.cc:1671] finished
0808 20:14:31.495713 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:31.496044 (+ 331us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16355,"raft.queue_time_us":115992,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"spinlock_wait_cycles":29688576,"thread_start_us":206410,"threads_started":2,"wal-append.queue_time_us":66302}
W20260808 20:14:31.652303 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 60757.5us
I20260808 20:14:31.735314 13016 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:31.588883 13072 rpcz_store.cc:276] 0808 20:14:29.780551 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.781932 (+ 1381us) service_pool.cc:227] Handling call
0808 20:14:31.220544 (+1438612us) spinlock_profiling.cc:243] Waited 4.84 ms on lock 0x7b5800005198. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537abce 000000000034aea9 00007fa96d1a6a30 00007fa96d26c7b4 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:31.220713 (+ 169us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:31.498321 (+277608us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:31.513443 (+ 15122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11140352}
W20260808 20:14:31.739980 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 118805us
W20260808 20:14:31.703117 12105 connection.cc:232] RPC call timeout handler was delayed by -1.5453492938540876s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1696s for a response
I20260808 20:14:31.680070 12249 consensus_queue.cc:260] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:31.683705 13073 rpcz_store.cc:276] 0808 20:14:28.897295 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:28.903377 (+ 6082us) service_pool.cc:227] Handling call
0808 20:14:28.921270 (+ 17893us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:30.338986 (+1417716us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:31.630604 (+1291618us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b540076edd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:31.632887 (+ 2283us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3259221888}
I20260808 20:14:31.705269 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:31.705780 15107 raft_consensus.cc:3037] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:31.682603 15210 leader_election.cc:288] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:31.703245 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:31.678849 15140 leader_election.cc:288] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:31.682700 15212 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:31.680102 15243 raft_consensus.cc:2781] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:31.727231 15109 raft_consensus.cc:2726] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:31.738220 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:31.680388 15142 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:31.738977 15308 log.cc:924] Time spent T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 1.230s user 0.004s sys 0.000s
I20260808 20:14:31.771656 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:31.777868 12247 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1799) took 2022 ms. Trace:
I20260808 20:14:31.790323 12247 rpcz_store.cc:276] 0808 20:14:29.755024 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:30.092224 (+337200us) service_pool.cc:227] Handling call
0808 20:14:31.015970 (+923746us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:31.075392 (+ 59422us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:31.777670 (+702278us) spinlock_profiling.cc:243] Waited 922 ms on lock 0x7b54001b0b50. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d9a09 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:31.777807 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2119845632}
I20260808 20:14:31.726359 15198 consensus_queue.cc:1046] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
W20260808 20:14:31.741253 12105 connection.cc:232] RPC call timeout handler was delayed by -1.4949492940213531s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17520000000000002s for a response
I20260808 20:14:31.752750 12206 leader_election.cc:302] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:31.707448 15285 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.684s user 0.022s sys 0.065s
I20260808 20:14:31.738916 15271 consensus_queue.cc:260] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:31.755213 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:31.816442 13078 raft_consensus.cc:2454] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:31.817548 13078 raft_consensus.cc:1226] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:31.817801 14987 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:31.818759 13078 raft_consensus.cc:1226] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 7. Current term is 8. Ops: []
I20260808 20:14:31.757751 12102 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
W20260808 20:14:31.819733 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 62087.1us
I20260808 20:14:31.759433 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1840) took 1201 ms. Trace:
I20260808 20:14:31.730373 15129 raft_consensus.cc:2726] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:31.683624 15281 raft_consensus.cc:504] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:31.766196 15361 raft_consensus.cc:2781] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260808 20:14:31.831761 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 79105.5us
I20260808 20:14:31.779060 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:31.767419 15108 consensus_queue.cc:1035] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 20:14:31.764595 15243 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:14:31.808059 12105 connection.cc:232] RPC call timeout handler was delayed by -1.2654492938891053s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20070000000000002s for a response
W20260808 20:14:31.815742 15285 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.811s user 0.023s sys 0.065s
I20260808 20:14:31.786562 15358 raft_consensus.cc:482] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:31.820924 12038 rpcz_store.cc:276] 0808 20:14:30.557884 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:30.560429 (+ 2545us) service_pool.cc:227] Handling call
0808 20:14:30.745287 (+184858us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:30.747408 (+ 2121us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:30.747436 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:30.747827 (+ 391us) log.cc:842] Serialized 26 byte log entry
0808 20:14:30.843232 (+ 95405us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:30.843265 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:30.843324 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:31.637037 (+793713us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:31.739972 (+102935us) raft_consensus.cc:1671] finished
0808 20:14:31.739991 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:31.759372 (+ 19381us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23604992,"thread_start_us":94207,"threads_started":1,"wal-append.queue_time_us":192311}
I20260808 20:14:31.758647 15269 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:31.766271 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:31.766462 15027 consensus_queue.cc:1035] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260808 20:14:31.780181 13074 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 649) took 3162 ms (client timeout 1752 ms). Trace:
I20260808 20:14:31.833245 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:31.833359 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:31.836607 12105 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:31.823917 12249 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:31.842123 12248 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 340) took 2097 ms. Trace:
I20260808 20:14:31.813791 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:31.841022 15219 raft_consensus.cc:2726] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 20:14:31.888116 13078 raft_consensus.cc:2454] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
W20260808 20:14:31.889722 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 53198.5us
I20260808 20:14:31.833498 15245 raft_consensus.cc:2781] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:31.899955 15245 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:31.849136 15243 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:31.890669 12105 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:31.891542 12207 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:31.886821 15163 raft_consensus.cc:482] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:31.841372 15285 raft_consensus.cc:686] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:31.821929 15287 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:31.897583 15358 raft_consensus.cc:504] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:31.856391 13074 rpcz_store.cc:269] 0808 20:14:28.617758 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:28.619666 (+ 1908us) service_pool.cc:227] Handling call
0808 20:14:31.779895 (+3160229us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b54006ef354. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:31.780070 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3281862528}
I20260808 20:14:31.924015 15100 consensus_queue.cc:260] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:31.934773 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:31.935659 12150 raft_consensus.cc:2396] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260808 20:14:31.936467 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:31.904145 15163 raft_consensus.cc:504] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:31.848278 15361 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:31.856796 14987 consensus_queue.cc:1046] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.052s
I20260808 20:14:31.909883 15236 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:31.856637 15098 consensus_queue.cc:1046] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260808 20:14:31.951673 15098 consensus_queue.cc:1225] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer e5bd9a4616ce4acfa20dc7a002984a20 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260808 20:14:31.914424 13071 raft_consensus.cc:2379] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 2: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:31.900379 15245 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:31.888715 15109 consensus_queue.cc:1046] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260808 20:14:31.915970 12105 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:31.960083 12105 leader_election.cc:302] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:31.923997 15204 raft_consensus.cc:2726] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:31.923204 15129 raft_consensus.cc:2726] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 20:14:31.902452 15198 raft_consensus.cc:482] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:31.968245 15292 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:31.972359 15198 raft_consensus.cc:504] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:31.973337 15163 leader_election.cc:288] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:31.902712 15142 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:31.858541 15269 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:31.902931 15255 raft_consensus.cc:2726] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 20:14:31.930984 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:31.937397 12150 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:31.946064 15361 raft_consensus.cc:3037] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:31.887274 12248 rpcz_store.cc:276] 0808 20:14:29.745134 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.745241 (+ 107us) service_pool.cc:227] Handling call
0808 20:14:29.745506 (+ 265us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:29.745805 (+ 299us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:29.745830 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:29.780022 (+ 34192us) log.cc:842] Serialized 26 byte log entry
0808 20:14:30.549234 (+769212us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:30.549275 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:30.549347 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:31.650928 (+1101581us) spinlock_profiling.cc:243] Waited 8.29 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:31.773017 (+122089us) raft_consensus.cc:1671] finished
0808 20:14:31.773038 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:31.842046 (+ 69008us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":397,"raft.queue_time_us":8135,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":9611,"spinlock_wait_cycles":19056000,"thread_start_us":718920,"threads_started":1,"wal-append.queue_time_us":692135}
I20260808 20:14:31.945479 15100 raft_consensus.cc:3037] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:31.923482 15234 raft_consensus.cc:2781] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:31.953495 15193 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:31.950975 12138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:53162 (request call id 1306) took 1787 ms. Trace:
I20260808 20:14:31.938819 13073 raft_consensus.cc:2379] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 8: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:31.961690 12105 leader_election.cc:302] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:31.957769 15098 raft_consensus.cc:3032] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:31.971879 15123 raft_consensus.cc:2726] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:31.979450 15198 leader_election.cc:288] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:31.949640 15358 leader_election.cc:288] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:31.974706 15292 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:31.993445 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:31.994407 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:31.984265 13072 raft_consensus.cc:2379] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:31.966185 15096 raft_consensus.cc:2726] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:14:31.995419 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:31.995916 12105 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:31.967808 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
W20260808 20:14:31.996559 12142 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.884s user 0.013s sys 0.018s
I20260808 20:14:31.933130 15337 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:32.005791 13078 raft_consensus.cc:2454] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
W20260808 20:14:31.989101 15051 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.977s user 0.052s sys 0.053s
I20260808 20:14:31.984418 15349 raft_consensus.cc:482] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:31.998642 15234 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:31.996917 15255 raft_consensus.cc:2726] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:31.969847 15164 consensus_queue.cc:1046] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
W20260808 20:14:31.997304 15278 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.055s user 0.011s sys 0.044s
I20260808 20:14:32.010072 12249 raft_consensus.cc:1261] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:14:32.015522 12105 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
W20260808 20:14:32.017839 12142 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.905s user 0.014s sys 0.018s
I20260808 20:14:32.019642 12207 leader_election.cc:302] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
W20260808 20:14:32.010279 12205 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.922s user 0.010s sys 0.024s
I20260808 20:14:31.994212 12138 rpcz_store.cc:276] 0808 20:14:30.169783 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:30.223668 (+ 53885us) service_pool.cc:227] Handling call
0808 20:14:31.950806 (+1727138us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b5400017554. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d1da7 00007fa96d21dee0 00007fa96690f784 00007fa96690f712 00007fa96690f641 00007fa96690f5bc 00007fa96690e872 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:31.950914 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":421313408}
I20260808 20:14:32.019321 14902 raft_consensus.cc:2726] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:32.021615 15246 raft_consensus.cc:2781] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260808 20:14:32.021992 15051 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.010s user 0.052s sys 0.053s
I20260808 20:14:31.996694 15098 raft_consensus.cc:729] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:32.044920 12146 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 7.
W20260808 20:14:32.042091 15278 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.117s user 0.011s sys 0.045s
W20260808 20:14:32.039458 12205 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.951s user 0.011s sys 0.024s
W20260808 20:14:32.018486 15152 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.198s user 0.020s sys 0.075s
I20260808 20:14:32.007958 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:32.034349 15312 log.cc:924] Time spent T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 1.433s user 0.004s sys 0.000s
I20260808 20:14:32.034550 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:32.043421 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:32.074880 12150 raft_consensus.cc:3037] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:32.046257 12147 raft_consensus.cc:2454] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:32.047705 12206 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:32.053470 15123 raft_consensus.cc:3032] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:32.037534 15234 raft_consensus.cc:3037] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
W20260808 20:14:32.056699 12103 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.016s sys 0.011s
W20260808 20:14:32.067389 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 978909us
W20260808 20:14:32.051950 15206 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.759s user 0.027s sys 0.022s
I20260808 20:14:32.052621 15360 raft_consensus.cc:2781] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:32.041654 15246 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:32.077056 12206 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:32.068091 15152 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.255s user 0.020s sys 0.075s
W20260808 20:14:32.035964 12045 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.899s user 0.020s sys 0.078s
I20260808 20:14:32.052027 15255 raft_consensus.cc:2781] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election won for term 3
W20260808 20:14:32.084278 15206 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.793s user 0.027s sys 0.022s
W20260808 20:14:32.083843 12103 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.676s user 0.017s sys 0.012s
W20260808 20:14:32.067853 12000 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.021s sys 0.035s
W20260808 20:14:32.100489 12000 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.667s user 0.021s sys 0.035s
W20260808 20:14:32.101070 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 667500us
I20260808 20:14:32.077419 12041 raft_consensus.cc:2454] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:32.102412 12000 leader_election.cc:302] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:32.078862 15123 raft_consensus.cc:729] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:32.087007 15360 raft_consensus.cc:482] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:32.053398 14902 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:32.078651 15109 raft_consensus.cc:2726] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:32.037844 15349 raft_consensus.cc:504] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:32.093442 12045 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.957s user 0.020s sys 0.078s
I20260808 20:14:32.109385 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:32.089109 15210 raft_consensus.cc:482] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:32.100809 15286 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260808 20:14:32.094579 15246 raft_consensus.cc:3037] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:32.096251 15108 raft_consensus.cc:2726] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:32.096376 15129 raft_consensus.cc:482] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:32.119109 15096 raft_consensus.cc:482] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:32.120713 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1604) took 1074 ms. Trace:
I20260808 20:14:32.091364 15358 consensus_queue.cc:1046] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260808 20:14:32.117199 15278 raft_consensus.cc:2781] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:14:32.036172 15285 consensus_queue.cc:237] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:32.080763 15098 consensus_queue.cc:260] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:32.103278 15193 raft_consensus.cc:482] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:32.145884 12142 rpcz_store.cc:276] 0808 20:14:31.046677 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:31.047270 (+ 593us) service_pool.cc:227] Handling call
0808 20:14:31.062345 (+ 15075us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:31.071667 (+ 9322us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:31.071698 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:31.082437 (+ 10739us) log.cc:842] Serialized 26 byte log entry
0808 20:14:32.038572 (+956135us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:32.038612 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:32.038701 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:32.120477 (+ 81776us) raft_consensus.cc:1671] finished
0808 20:14:32.120500 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:32.120654 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9034,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":104,"thread_start_us":925814,"threads_started":1,"wal-append.queue_time_us":999270}
I20260808 20:14:32.149775 12150 raft_consensus.cc:2454] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:32.108639 15255 raft_consensus.cc:686] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:32.151647 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:32.155794 12047 raft_consensus.cc:3037] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:32.126160 14777 raft_consensus.cc:482] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:32.156330 13016 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:32.118331 15360 raft_consensus.cc:3037] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:32.140576 15191 raft_consensus.cc:3032] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:32.127295 15204 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:32.161190 13015 leader_election.cc:302] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:32.108204 15339 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:32.127600 15164 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:32.162497 15269 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:32.158032 15193 raft_consensus.cc:504] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:32.163652 14777 raft_consensus.cc:504] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:32.147190 12041 raft_consensus.cc:1226] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:32.168237 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:32.170714 12142 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:32.166056 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:32.154830 13073 raft_consensus.cc:1226] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:32.126969 15210 raft_consensus.cc:504] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:32.185400 12150 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:32.155104 15337 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:32.121275 14902 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:32.188433 12046 raft_consensus.cc:1226] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:32.154755 15278 raft_consensus.cc:686] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:32.190263 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:32.167362 13075 raft_consensus.cc:1261] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 20:14:32.169519 15163 raft_consensus.cc:2781] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:32.190464 13016 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:32.158397 15191 raft_consensus.cc:729] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:32.165649 15369 raft_consensus.cc:2781] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:32.149713 15129 raft_consensus.cc:504] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:32.172459 15352 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:32.149379 15096 raft_consensus.cc:504] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:32.201097 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:32.164765 15243 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:32.206245 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:32.195859 15269 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:32.208371 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:32.150604 15123 consensus_queue.cc:260] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:32.169914 15164 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:32.211490 11994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.773s user 0.015s sys 0.017s
I20260808 20:14:32.201951 15163 raft_consensus.cc:482] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:14:32.206615 12137 consensus_meta.cc:287] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.585s user 0.000s sys 0.012s
I20260808 20:14:32.169330 15204 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:32.211676 12101 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.777s user 0.017s sys 0.031s
I20260808 20:14:32.204753 15287 raft_consensus.cc:2781] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:32.220371 13016 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:32.224090 12101 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.796s user 0.018s sys 0.032s
W20260808 20:14:32.224857 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 797715us
I20260808 20:14:32.225765 12101 leader_election.cc:302] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:32.226595 15143 raft_consensus.cc:2741] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260808 20:14:32.226999 12101 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:32.227766 15143 raft_consensus.cc:2781] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:32.212625 15352 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:32.218353 11994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.780s user 0.015s sys 0.017s
W20260808 20:14:32.211719 13014 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.745s user 0.020s sys 0.011s
W20260808 20:14:32.407804 13014 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.941s user 0.020s sys 0.011s
I20260808 20:14:32.230666 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:32.408284 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 1.07678e+06us
I20260808 20:14:32.228113 12101 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:32.409320 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 181252us
I20260808 20:14:32.410352 12101 leader_election.cc:302] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:32.265838 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 219531us
I20260808 20:14:32.223453 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:32.228485 15108 raft_consensus.cc:482] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:32.208447 15198 consensus_queue.cc:1046] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260808 20:14:32.228171 15143 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:32.234133 15278 consensus_queue.cc:237] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:32.318295 15123 raft_consensus.cc:3037] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
W20260808 20:14:32.229447 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 901409us
W20260808 20:14:32.211637 15342 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.153s user 0.005s sys 0.044s
I20260808 20:14:32.223562 15287 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:32.222918 15163 raft_consensus.cc:3037] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:32.230392 15191 consensus_queue.cc:260] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:32.222805 12137 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:14:32.212730 15255 consensus_queue.cc:237] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:32.229094 15027 consensus_queue.cc:1046] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.109s
I20260808 20:14:32.218366 15269 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:32.415334 12101 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:32.444061 15198 consensus_queue.cc:1225] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer 4cd46f13bceb4c3c9ef71e755e17b995 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260808 20:14:32.428572 12047 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:14:32.445063 11994 leader_election.cc:302] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:32.446240 15342 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.388s user 0.006s sys 0.054s
I20260808 20:14:32.450344 15287 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:32.477463 12041 raft_consensus.cc:2379] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 5: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:32.434854 15143 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:32.453123 15286 raft_consensus.cc:3032] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 7 LEADER]: Stepping down as leader of term 7
W20260808 20:14:32.425022 15366 log.cc:924] Time spent T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.256s user 0.007s sys 0.000s
W20260808 20:14:32.434209 12141 consensus_meta.cc:287] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.583s user 0.013s sys 0.005s
I20260808 20:14:32.417716 15364 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:32.422559 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 202281us
I20260808 20:14:32.441370 15339 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:32.417716 15375 raft_consensus.cc:2726] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:14:32.523242 13079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53296 (request call id 654) took 2025 ms. Trace:
I20260808 20:14:32.526912 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:32.527204 12205 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:32.461589 13073 raft_consensus.cc:1261] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEARNER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260808 20:14:32.480329 15369 raft_consensus.cc:686] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:32.460184 15374 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:32.445010 15361 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:32.432945 15108 raft_consensus.cc:504] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:32.567902 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 152633us
I20260808 20:14:32.568945 13079 rpcz_store.cc:276] 0808 20:14:30.497408 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:31.073421 (+576013us) service_pool.cc:227] Handling call
0808 20:14:31.076566 (+ 3145us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:31.460357 (+383791us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:32.522981 (+1062624us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b5400524ed4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:32.523175 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":822157696}
I20260808 20:14:32.489028 15384 raft_consensus.cc:2726] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:32.568658 12047 raft_consensus.cc:2454] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:32.531831 15191 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
W20260808 20:14:32.571029 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 43931.3us
I20260808 20:14:32.541386 15161 raft_consensus.cc:2726] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 20:14:32.488833 12137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1607) took 1430 ms. Trace:
I20260808 20:14:32.575903 12101 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:32.674966 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 99155.9us
I20260808 20:14:32.545368 15264 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:32.582988 15361 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:32.552457 12141 raft_consensus.cc:2454] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:32.583770 15210 leader_election.cc:288] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:32.585274 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:32.572690 15243 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:32.545280 15286 raft_consensus.cc:729] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:32.570138 14902 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:32.570370 15129 leader_election.cc:288] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:32.553037 15342 raft_consensus.cc:3037] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:32.570796 15193 leader_election.cc:288] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:32.572177 14777 leader_election.cc:288] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:32.685122 12249 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:14:32.747818 12249 raft_consensus.cc:595] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:32.748188 12249 raft_consensus.cc:603] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 20:14:32.749250 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:32.574283 15027 raft_consensus.cc:2781] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:32.586341 15236 raft_consensus.cc:3032] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:32.574165 15164 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:32.586293 15108 leader_election.cc:288] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:32.749868 12249 raft_consensus.cc:2454] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:32.591007 15283 raft_consensus.cc:482] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:32.715617 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:32.588737 15096 leader_election.cc:288] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:32.719139 12205 leader_election.cc:302] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:32.720831 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:32.754190 12247 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 7.
I20260808 20:14:32.741099 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:32.755268 12047 raft_consensus.cc:2454] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:32.743719 12045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 429) took 1791 ms. Trace:
I20260808 20:14:32.756304 12045 rpcz_store.cc:276] 0808 20:14:30.951910 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:30.984109 (+ 32199us) service_pool.cc:227] Handling call
0808 20:14:31.012680 (+ 28571us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 20:14:31.077886 (+ 65206us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:31.077916 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 20:14:31.081415 (+ 3499us) log.cc:842] Serialized 52 byte log entry
0808 20:14:32.123369 (+1041954us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:32.123410 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:32.123489 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:32.743409 (+619920us) raft_consensus.cc:1671] finished
0808 20:14:32.743430 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:32.743650 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8301,"raft.queue_time_us":6784,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":4784,"thread_start_us":986857,"threads_started":1,"wal-append.queue_time_us":1060155}
I20260808 20:14:32.685391 15375 raft_consensus.cc:2781] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:32.735316 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:32.574100 15374 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:32.572816 15364 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:32.703423 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:32.673717 12137 rpcz_store.cc:276] 0808 20:14:31.050590 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:31.052245 (+ 1655us) service_pool.cc:227] Handling call
0808 20:14:32.481285 (+1429040us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:32.788107 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:32.800115 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:32.790024 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:32.800851 12045 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
W20260808 20:14:32.747534 15234 consensus_meta.cc:287] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.660s user 0.008s sys 0.001s
I20260808 20:14:32.792079 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:32.747565 14777 raft_consensus.cc:482] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:32.873478 15269 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:32.873342 14777 raft_consensus.cc:504] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:32.876137 14777 leader_election.cc:288] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:32.742345 15129 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:32.752252 15401 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:32.686697 15264 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:32.798087 15343 raft_consensus.cc:3032] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:32.877247 15343 raft_consensus.cc:729] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:32.876919 15129 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:32.878098 15343 consensus_queue.cc:260] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:32.879123 15343 raft_consensus.cc:3037] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:32.774437 15283 raft_consensus.cc:504] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:32.880406 15129 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:32.881191 12249 raft_consensus.cc:2454] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:32.872365 15400 consensus_queue.cc:1046] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 20:14:32.736222 15408 raft_consensus.cc:2726] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 20:14:32.881544 15401 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:32.881932 15283 leader_election.cc:288] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:14:32.871618 15360 consensus_meta.cc:287] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.702s user 0.004s sys 0.005s
I20260808 20:14:32.883116 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:32.758247 15236 raft_consensus.cc:729] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:32.758021 15027 raft_consensus.cc:686] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
W20260808 20:14:32.920995 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 86205.7us
I20260808 20:14:32.801298 12248 raft_consensus.cc:2454] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
W20260808 20:14:32.819532 12152 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 284) took 1979 ms (client timeout 1639 ms). Trace:
W20260808 20:14:32.891927 12040 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49226 (request call id 433) took 1512 ms (client timeout 1564 ms). Trace:
W20260808 20:14:32.898201 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 151175us
I20260808 20:14:32.773579 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:32.781080 15108 raft_consensus.cc:2726] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:32.923795 12000 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
W20260808 20:14:32.887312 15100 consensus_meta.cc:287] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.885s user 0.007s sys 0.003s
W20260808 20:14:32.926159 12040 rpcz_store.cc:269] 0808 20:14:31.307247 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:31.472009 (+164762us) service_pool.cc:227] Handling call
0808 20:14:32.819917 (+1347908us) spinlock_profiling.cc:243] Waited 64.8 ms on lock 0x7b540009ea54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:32.820051 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":149095680}
W20260808 20:14:32.813383 15385 log.cc:924] Time spent T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.393s user 0.001s sys 0.000s
I20260808 20:14:32.931741 13014 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:32.935266 12150 raft_consensus.cc:2454] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
W20260808 20:14:32.937419 15245 consensus_meta.cc:287] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.962s user 0.009s sys 0.003s
I20260808 20:14:32.917335 15283 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:32.927111 15400 raft_consensus.cc:3032] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:32.938364 15236 consensus_queue.cc:260] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:32.939987 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:32.941241 15273 log.cc:924] Time spent T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 2.860s user 0.001s sys 0.016s
I20260808 20:14:32.941378 15264 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:32.942421 15206 raft_consensus.cc:3032] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:32.942005 15027 consensus_queue.cc:237] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:32.885636 12041 raft_consensus.cc:2454] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:32.944868 15374 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:32.945899 15143 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:32.946985 15364 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:32.948339 12141 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1313) took 1623 ms (client timeout 1514 ms). Trace:
I20260808 20:14:32.949476 15401 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:32.951222 15140 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:32.951582 15330 log.cc:924] Time spent T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 2.012s user 0.008s sys 0.017s
W20260808 20:14:32.951956 15123 consensus_meta.cc:287] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.501s user 0.008s sys 0.000s
I20260808 20:14:32.952190 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1371) took 2016 ms. Trace:
I20260808 20:14:32.952932 13080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 422) took 2171 ms. Trace:
W20260808 20:14:32.953536 15389 log.cc:924] Time spent T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.376s user 0.001s sys 0.000s
W20260808 20:14:32.953680 15318 log.cc:924] Time spent T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045: Append to log took a long time: real 2.036s user 0.004s sys 0.009s
W20260808 20:14:32.954038 15303 log.cc:924] Time spent T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 2.449s user 0.009s sys 0.004s
I20260808 20:14:32.954520 13079 raft_consensus.cc:1226] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:32.944864 13075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 421) took 2178 ms. Trace:
I20260808 20:14:32.931816 15125 raft_consensus.cc:482] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:32.956271 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:32.935031 15337 raft_consensus.cc:2781] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:32.954097 13072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 658) took 1253 ms. Trace:
W20260808 20:14:32.959893 15246 consensus_meta.cc:287] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.814s user 0.008s sys 0.000s
I20260808 20:14:32.962875 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:32.964346 15098 raft_consensus.cc:3037] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:33.020304 15287 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:32.966923 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:32.967331 15205 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:32.952087 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:32.970140 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:32.932264 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:32.973299 12048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 426) took 2047 ms. Trace:
I20260808 20:14:32.973834 15352 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:32.974944 12152 rpcz_store.cc:269] 0808 20:14:30.840136 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:30.840295 (+ 159us) service_pool.cc:227] Handling call
0808 20:14:32.819351 (+1979056us) spinlock_profiling.cc:243] Waited 74.6 ms on lock 0x7b540017f554. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:32.819461 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":171653376}
I20260808 20:14:32.979394 12045 raft_consensus.cc:2454] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 7.
I20260808 20:14:32.809829 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:32.792949 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:33.003957 12101 leader_election.cc:302] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:33.006513 12102 leader_election.cc:302] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:32.917304 15198 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
W20260808 20:14:33.009574 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 77922.3us
I20260808 20:14:32.842746 15375 raft_consensus.cc:686] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:32.972110 15204 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:33.020007 12248 raft_consensus.cc:1261] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260808 20:14:32.775760 15243 consensus_queue.cc:1035] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.259s
I20260808 20:14:33.005445 15400 raft_consensus.cc:729] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
W20260808 20:14:33.028992 15163 consensus_meta.cc:287] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.548s user 0.008s sys 0.000s
I20260808 20:14:33.010172 15403 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:33.018217 15408 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:33.018025 15206 raft_consensus.cc:729] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:33.041165 12142 raft_consensus.cc:2454] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:33.024442 15107 raft_consensus.cc:2781] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260808 20:14:33.038653 15192 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.500s user 0.012s sys 0.009s
I20260808 20:14:32.794327 15286 consensus_queue.cc:260] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:33.061339 13015 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.013s sys 0.036s
I20260808 20:14:33.041018 13080 rpcz_store.cc:276] 0808 20:14:30.781641 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:31.467734 (+686093us) service_pool.cc:227] Handling call
0808 20:14:31.468054 (+ 320us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:31.643849 (+175795us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:31.643878 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:31.643928 (+ 50us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:31.643963 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:32.952688 (+1308725us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b540069f354. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:32.952728 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:32.952860 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":403834752}
I20260808 20:14:33.032353 15236 raft_consensus.cc:3037] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:32.780436 15369 consensus_queue.cc:237] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:33.011600 15283 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:33.073311 12244 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:14:32.920471 15360 raft_consensus.cc:504] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:33.005380 15245 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:33.090899 12105 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.008s sys 0.012s
W20260808 20:14:33.091012 11994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.007s sys 0.013s
I20260808 20:14:32.930879 15234 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:33.089854 15227 raft_consensus.cc:482] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:33.098141 15143 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:33.091486 15246 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:33.102321 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:33.503842 13015 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.972s user 0.038s sys 0.088s
I20260808 20:14:33.071851 13075 rpcz_store.cc:276] 0808 20:14:30.766766 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:31.097296 (+330530us) service_pool.cc:227] Handling call
0808 20:14:31.235612 (+138316us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:32.202682 (+967070us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:32.944495 (+741813us) spinlock_profiling.cc:243] Waited 894 ms on lock 0x7b5400521cd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:32.944781 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2055800064}
I20260808 20:14:33.039263 15140 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:33.518330 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 514434us
W20260808 20:14:33.115324 12251 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1821) took 1767 ms (client timeout 1664 ms). Trace:
W20260808 20:14:33.119192 12245 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1796) took 3365 ms (client timeout 1732 ms). Trace:
I20260808 20:14:33.088665 15100 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:33.503743 13078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 430) took 1574 ms. Trace:
I20260808 20:14:33.522223 13078 rpcz_store.cc:276] 0808 20:14:31.928884 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:32.690803 (+761919us) service_pool.cc:227] Handling call
0808 20:14:32.691143 (+ 340us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:32.716784 (+ 25641us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:32.716818 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:32.736017 (+ 19199us) log.cc:842] Serialized 26 byte log entry
0808 20:14:33.005980 (+269963us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:33.006018 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:33.006082 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:33.503531 (+497449us) raft_consensus.cc:1671] finished
0808 20:14:33.503551 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:33.503688 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":179466,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":184,"thread_start_us":225558,"threads_started":1,"wal-append.queue_time_us":336516}
I20260808 20:14:33.523506 13078 raft_consensus.cc:1226] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6. Ops: []
W20260808 20:14:33.489313 12206 leader_election.cc:334] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.569s (SENT)
I20260808 20:14:33.116714 12046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1849) took 1530 ms. Trace:
I20260808 20:14:33.506407 12045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 1856) took 1027 ms. Trace:
I20260808 20:14:33.524601 12045 rpcz_store.cc:276] 0808 20:14:32.479413 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:32.761518 (+282105us) service_pool.cc:227] Handling call
0808 20:14:33.506349 (+744831us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:33.504989 12105 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.015s user 0.008s sys 0.022s
W20260808 20:14:33.504347 11994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.978s user 0.013s sys 0.017s
I20260808 20:14:33.508190 12047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49226 (request call id 440) took 1347 ms. Trace:
I20260808 20:14:33.525305 12046 rpcz_store.cc:276] 0808 20:14:31.586315 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:31.830552 (+244237us) service_pool.cc:227] Handling call
0808 20:14:31.859911 (+ 29359us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:32.193905 (+333994us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:33.116514 (+922609us) spinlock_profiling.cc:243] Waited 328 ms on lock 0x7b5400465ad4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:33.116663 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":755078912}
I20260808 20:14:33.525894 12047 rpcz_store.cc:276] 0808 20:14:32.160785 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:32.775559 (+614774us) service_pool.cc:227] Handling call
0808 20:14:33.508135 (+732576us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:33.526968 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 1.0819e+06us
I20260808 20:14:33.505780 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:33.029559 12141 rpcz_store.cc:269] 0808 20:14:31.324421 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:31.497066 (+172645us) service_pool.cc:227] Handling call
0808 20:14:32.948089 (+1451023us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b5400017054. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:32.948242 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":592532352}
I20260808 20:14:33.506578 12244 raft_consensus.cc:595] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:33.509603 12001 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.24.4:36901 timed out after 1.860s (SENT) [suppressed 38 similar messages]
I20260808 20:14:33.118772 12250 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 355) took 1646 ms. Trace:
I20260808 20:14:33.528738 12152 raft_consensus.cc:2379] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 5: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
W20260808 20:14:33.514842 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 508360us
I20260808 20:14:33.378288 15117 raft_consensus.cc:482] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:33.100001 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:33.126652 15281 leader_election.cc:288] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:33.535604 12247 raft_consensus.cc:2454] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
W20260808 20:14:33.503782 15192 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.981s user 0.014s sys 0.017s
I20260808 20:14:33.549582 12205 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:33.516404 12243 raft_consensus.cc:3037] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:33.517800 12043 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1365) took 3011 ms. Trace:
I20260808 20:14:33.521418 12101 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:33.524185 12206 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:33.570466 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 81190.8us
W20260808 20:14:33.089413 13071 consensus_meta.cc:287] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 1.054s user 0.011s sys 0.001s
W20260808 20:14:33.369931 15342 consensus_meta.cc:287] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.629s user 0.010s sys 0.000s
I20260808 20:14:33.116235 15333 raft_consensus.cc:3032] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:33.091470 15337 raft_consensus.cc:686] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:33.446259 15210 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:33.527946 12250 rpcz_store.cc:276] 0808 20:14:31.472476 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:31.636854 (+164378us) service_pool.cc:227] Handling call
0808 20:14:31.637504 (+ 650us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 20:14:31.773571 (+136067us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:31.773598 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 20:14:31.778319 (+ 4721us) log.cc:842] Serialized 52 byte log entry
0808 20:14:32.160841 (+382522us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:32.160879 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:33.118385 (+957506us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b5400375d54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:33.118433 (+ 48us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:33.118478 (+ 45us) raft_consensus.cc:1671] finished
0808 20:14:33.118493 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:33.118712 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10461,"raft.queue_time_us":85,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":134,"spinlock_wait_cycles":312061184,"thread_start_us":381085,"threads_started":1,"wal-append.queue_time_us":445018}
I20260808 20:14:33.534075 12045 raft_consensus.cc:1226] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 2. Current term is 3. Ops: []
W20260808 20:14:33.519130 12245 rpcz_store.cc:269] 0808 20:14:29.754206 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.757169 (+ 2963us) service_pool.cc:227] Handling call
0808 20:14:33.119005 (+3361836us) spinlock_profiling.cc:243] Waited 1.82 s on lock 0x7b5400375854. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:33.119129 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4180412672}
I20260808 20:14:33.121168 15104 raft_consensus.cc:3032] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Stepping down as leader of term 1
W20260808 20:14:33.708709 12206 leader_election.cc:334] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.821s (SENT)
W20260808 20:14:33.536653 12152 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1321) took 2038 ms (client timeout 2289 ms). Trace:
I20260808 20:14:33.446429 15289 raft_consensus.cc:482] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:33.503911 15202 raft_consensus.cc:3037] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
W20260808 20:14:33.519748 12251 rpcz_store.cc:269] 0808 20:14:31.348166 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:31.368483 (+ 20317us) service_pool.cc:227] Handling call
0808 20:14:33.115114 (+1746631us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b540029d954. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:33.115247 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3741471104}
I20260808 20:14:33.811757 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:33.481142 15322 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
W20260808 20:14:33.625938 15303 log.cc:924] Time spent T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260808 20:14:33.507835 15237 raft_consensus.cc:3037] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:33.114042 15311 raft_consensus.cc:3032] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:33.094133 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:33.507866 15107 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:33.503126 12137 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 7.
I20260808 20:14:33.088367 15125 raft_consensus.cc:504] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:33.552362 15193 raft_consensus.cc:482] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:33.531319 15204 raft_consensus.cc:2781] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:33.715067 13071 raft_consensus.cc:2454] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:33.076232 13072 rpcz_store.cc:276] 0808 20:14:31.700462 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:31.774518 (+ 74056us) service_pool.cc:227] Handling call
0808 20:14:32.953772 (+1179254us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b5400581454. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:32.954004 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":326216960}
I20260808 20:14:33.561182 12043 rpcz_store.cc:276] 0808 20:14:30.506608 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:30.506711 (+ 103us) service_pool.cc:227] Handling call
0808 20:14:30.507005 (+ 294us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:30.747722 (+240717us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:30.747752 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:30.748208 (+ 456us) log.cc:842] Serialized 26 byte log entry
0808 20:14:30.805421 (+ 57213us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:30.805464 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:30.805528 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:32.950725 (+2145197us) spinlock_profiling.cc:243] Waited 3.21 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:33.126237 (+175512us) raft_consensus.cc:1671] finished
0808 20:14:33.126256 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:33.517742 (+391486us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":216591,"raft.queue_time_us":228016,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":124,"spinlock_wait_cycles":7393024,"thread_start_us":56818,"threads_started":1,"wal-append.queue_time_us":190900}
I20260808 20:14:33.529407 15096 raft_consensus.cc:2781] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:33.513798 15227 raft_consensus.cc:504] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:33.040856 12036 rpcz_store.cc:276] 0808 20:14:30.935328 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:30.976006 (+ 40678us) service_pool.cc:227] Handling call
0808 20:14:30.976297 (+ 291us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:31.061710 (+ 85413us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:32.951825 (+1890115us) spinlock_profiling.cc:243] Waited 5.8 ms on lock 0x7b540034d854. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:32.952101 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13339008}
W20260808 20:14:33.636605 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 66986.9us
I20260808 20:14:33.516149 15198 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:33.860558 15098 consensus_meta.cc:287] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.768s user 0.007s sys 0.002s
I20260808 20:14:33.126287 15205 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:33.882359 15434 log.cc:924] Time spent T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.254s user 0.000s sys 0.001s
I20260808 20:14:33.883391 15205 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:33.515938 15403 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:33.515249 15163 raft_consensus.cc:504] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:33.894125 15163 leader_election.cc:288] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:33.894625 15219 raft_consensus.cc:482] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:33.895262 15219 raft_consensus.cc:504] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:33.605717 15285 raft_consensus.cc:2726] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:33.539672 15100 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:33.706656 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:33.661909 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 140548us
I20260808 20:14:33.614483 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:33.536041 11994 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:33.795079 12045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1375) took 1696 ms. Trace:
I20260808 20:14:33.562672 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:33.798266 12152 rpcz_store.cc:269] 0808 20:14:31.497848 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:33.505528 (+2007680us) service_pool.cc:227] Handling call
0808 20:14:33.536591 (+ 31063us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:33.797487 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 88831.9us
W20260808 20:14:33.910991 12206 leader_election.cc:334] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.714s (SENT)
I20260808 20:14:33.911433 12206 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:33.912567 12206 leader_election.cc:334] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.913s (SENT)
I20260808 20:14:33.509697 14987 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:33.819227 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:33.712663 15333 raft_consensus.cc:729] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
W20260808 20:14:33.615674 15427 log.cc:924] Time spent T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20260808 20:14:33.818137 12245 raft_consensus.cc:2362] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3. Re-sending same reply.
I20260808 20:14:33.919476 12245 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:14:33.919656 12101 leader_election.cc:302] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:33.920047 12245 raft_consensus.cc:595] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:33.486342 15287 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:33.713112 15426 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:33.834365 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:33.623718 15234 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:33.636078 15334 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:33.613660 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:33.617256 15142 raft_consensus.cc:2726] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 20:14:33.562779 12105 service_pool.cc:137] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:53148 dropped due to backpressure. The service queue is full; it has 50 items.
I20260808 20:14:33.617141 15245 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:33.837070 12137 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1332) took 2230 ms (client timeout 2158 ms). Trace:
I20260808 20:14:33.794663 15104 raft_consensus.cc:729] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:33.855059 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:33.794682 15342 raft_consensus.cc:2726] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:33.716773 15432 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:33.610054 15246 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:33.625550 15283 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:33.616022 15132 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:33.795559 15192 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:33.572649 12047 raft_consensus.cc:1226] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260808 20:14:33.961416 12047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1377) took 1852 ms. Trace:
I20260808 20:14:33.961738 12047 rpcz_store.cc:276] 0808 20:14:32.108624 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:33.533214 (+1424590us) service_pool.cc:227] Handling call
0808 20:14:33.572426 (+ 39212us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:33.961178 (+388752us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:33.961364 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:33.613056 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:33.627918 15140 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:33.504146 12048 rpcz_store.cc:276] 0808 20:14:30.925977 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:30.932968 (+ 6991us) service_pool.cc:227] Handling call
0808 20:14:30.960684 (+ 27716us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:31.577826 (+617142us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:32.973049 (+1395223us) spinlock_profiling.cc:243] Waited 600 ms on lock 0x7b540034e754. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:32.973216 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1379319168}
I20260808 20:14:33.819315 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:33.714087 15352 raft_consensus.cc:482] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:33.835244 15107 raft_consensus.cc:3037] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:33.830374 15311 raft_consensus.cc:729] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:33.550309 15117 raft_consensus.cc:504] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:33.569376 15364 raft_consensus.cc:482] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:33.572122 15206 consensus_queue.cc:260] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:33.879280 15187 raft_consensus.cc:3032] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:33.868571 15096 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:33.520885 15408 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:33.885238 15271 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:33.886121 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:33.886600 12150 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 292) took 2305 ms (client timeout 2180 ms). Trace:
W20260808 20:14:33.985984 12150 rpcz_store.cc:269] 0808 20:14:31.581633 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:32.219338 (+637705us) service_pool.cc:227] Handling call
0808 20:14:33.886444 (+1667106us) spinlock_profiling.cc:243] Waited 525 ms on lock 0x7b5400106154. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:33.886533 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1208198528}
I20260808 20:14:33.886755 12144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1599) took 5561 ms. Trace:
W20260808 20:14:33.887866 15343 consensus_meta.cc:287] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 1.008s user 0.009s sys 0.002s
I20260808 20:14:33.889029 15125 leader_election.cc:288] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:33.802295 15210 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:33.857506 15204 raft_consensus.cc:482] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:33.855062 15433 raft_consensus.cc:2781] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260808 20:14:33.916764 12206 leader_election.cc:334] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.914s (SENT)
W20260808 20:14:33.992702 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 75967.9us
I20260808 20:14:33.829278 15322 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:33.921373 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 385400us
I20260808 20:14:33.831063 15289 raft_consensus.cc:504] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:33.922102 12045 rpcz_store.cc:276] 0808 20:14:32.099054 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:33.533641 (+1434587us) service_pool.cc:227] Handling call
0808 20:14:33.533915 (+ 274us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:33.794791 (+260876us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:33.795023 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:33.923586 12250 raft_consensus.cc:2379] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
W20260808 20:14:33.871719 13071 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57330 (request call id 431) took 1918 ms (client timeout 1522 ms). Trace:
I20260808 20:14:33.927181 13078 raft_consensus.cc:2454] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:33.697252 15400 consensus_queue.cc:260] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:33.880707 15227 leader_election.cc:288] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:33.934953 12105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:53148 (request call id 1343) took 1781 ms. Trace:
W20260808 20:14:33.935825 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 173081us
I20260808 20:14:34.010879 12045 raft_consensus.cc:1226] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260808 20:14:33.860721 15193 raft_consensus.cc:504] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:33.935132 15164 raft_consensus.cc:2781] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:33.951503 15349 leader_election.cc:288] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:33.951930 12248 raft_consensus.cc:1226] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260808 20:14:33.953008 15269 raft_consensus.cc:482] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:33.954818 15360 leader_election.cc:288] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:33.955039 15243 consensus_queue.cc:1035] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
W20260808 20:14:33.955478 12146 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1308) took 2633 ms (client timeout 1821 ms). Trace:
W20260808 20:14:34.017050 12146 rpcz_store.cc:269] 0808 20:14:31.322461 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:31.514750 (+192289us) service_pool.cc:227] Handling call
0808 20:14:33.955294 (+2440544us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b54004254d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:33.955385 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":881829760}
I20260808 20:14:34.018137 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
W20260808 20:14:34.018862 12146 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1309) took 2696 ms (client timeout 2830 ms). Trace:
W20260808 20:14:34.019157 12146 rpcz_store.cc:269] 0808 20:14:31.322815 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.017892 (+2695077us) service_pool.cc:227] Handling call
0808 20:14:34.018810 (+ 918us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:34.019912 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
W20260808 20:14:34.020555 12146 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1623) took 1584 ms (client timeout 1976 ms). Trace:
I20260808 20:14:33.918318 15098 raft_consensus.cc:2726] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:33.926760 15278 raft_consensus.cc:2726] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:33.922128 13074 raft_consensus.cc:2454] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
W20260808 20:14:33.963328 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 453722us
I20260808 20:14:34.021723 13074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53290 (request call id 519) took 1443 ms. Trace:
I20260808 20:14:34.022079 13074 rpcz_store.cc:276] 0808 20:14:32.578635 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:33.098005 (+519370us) service_pool.cc:227] Handling call
0808 20:14:34.021654 (+923649us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:34.022967 15198 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:34.023149 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:33.962472 12040 raft_consensus.cc:2454] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:33.711860 12102 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53176 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1329)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1624)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1309)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1623)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 299)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1620)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1618)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 291)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 301)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1343)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1625)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1335)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1336)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1337)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1338)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 303)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1339)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1341)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1342)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 300)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1629)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1612)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1614)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 288)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 289)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1310)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1311)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1315)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1316)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1317)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1318)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1322)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1323)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1326)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 293)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 295)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 296)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1616)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1617)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1619)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1622)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 304)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 305)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1626)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1340)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1627)
I20260808 20:14:34.025784 15269 raft_consensus.cc:504] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:33.966153 12147 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1314) took 2641 ms (client timeout 1914 ms). Trace:
W20260808 20:14:34.026854 12147 rpcz_store.cc:269] 0808 20:14:31.324795 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:31.732297 (+407502us) service_pool.cc:227] Handling call
0808 20:14:33.965979 (+2233682us) spinlock_profiling.cc:243] Waited 277 ms on lock 0x7b54000609d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:33.966084 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":637687168}
I20260808 20:14:34.027869 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:34.028550 12147 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 299) took 1872 ms (client timeout 2378 ms). Trace:
I20260808 20:14:33.967320 12140 raft_consensus.cc:1261] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 20:14:34.028851 15452 consensus_queue.cc:1046] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 20:14:34.012460 15300 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:33.857904 15375 consensus_queue.cc:237] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:33.973546 12047 raft_consensus.cc:1226] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7. Ops: []
I20260808 20:14:34.030865 12047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1387) took 1832 ms. Trace:
I20260808 20:14:34.031167 12047 rpcz_store.cc:276] 0808 20:14:32.198292 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:33.962550 (+1764258us) service_pool.cc:227] Handling call
0808 20:14:33.973397 (+ 10847us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:34.030644 (+ 57247us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:34.030815 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:34.031690 15027 consensus_queue.cc:1046] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:14:34.032047 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:33.957263 15234 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:33.958166 15205 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:33.981817 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 274177us
I20260808 20:14:34.024506 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:33.969426 15285 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:33.935916 12137 rpcz_store.cc:269] 0808 20:14:31.607068 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:32.903833 (+1296765us) service_pool.cc:227] Handling call
0808 20:14:33.836981 (+933148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:33.929949 14987 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:33.897109 15337 consensus_queue.cc:237] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:33.984822 12141 raft_consensus.cc:2379] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 6: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:33.963477 15283 consensus_queue.cc:1035] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260808 20:14:33.976401 15187 raft_consensus.cc:729] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:33.933693 15334 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:34.014606 15164 raft_consensus.cc:482] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:33.994419 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:33.963227 15132 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:33.992136 15204 raft_consensus.cc:3037] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:14:33.995711 12206 leader_election.cc:334] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.983s (SENT)
I20260808 20:14:33.964133 15432 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:34.001430 13071 rpcz_store.cc:269] 0808 20:14:31.948666 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:31.967558 (+ 18892us) service_pool.cc:227] Handling call
0808 20:14:33.867412 (+1899854us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:33.984547 15104 consensus_queue.cc:260] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:34.009630 12105 rpcz_store.cc:276] 0808 20:14:32.148818 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:33.930601 (+1781783us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 20:14:33.973006 15352 raft_consensus.cc:504] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:34.046284 15288 raft_consensus.cc:3032] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:34.013466 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:34.015547 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1364) took 3515 ms. Trace:
I20260808 20:14:34.108696 13078 raft_consensus.cc:2362] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5. Re-sending same reply.
I20260808 20:14:34.108629 12039 rpcz_store.cc:276] 0808 20:14:30.500567 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:30.505939 (+ 5372us) service_pool.cc:227] Handling call
0808 20:14:30.506207 (+ 268us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:31.673205 (+1166998us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:34.015364 (+2342159us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54003b4554. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:34.015495 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2582074240}
I20260808 20:14:34.058022 12048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1388) took 1644 ms. Trace:
I20260808 20:14:34.029843 15300 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:34.032931 12047 raft_consensus.cc:3037] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
W20260808 20:14:34.028855 12147 rpcz_store.cc:269] 0808 20:14:32.155875 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.027643 (+1871768us) service_pool.cc:227] Handling call
0808 20:14:34.028504 (+ 861us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:33.981652 15364 raft_consensus.cc:504] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:34.020852 12146 rpcz_store.cc:269] 0808 20:14:32.435848 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.019695 (+1583847us) service_pool.cc:227] Handling call
0808 20:14:34.020512 (+ 817us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:34.035190 12102 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53176 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1620)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1618)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 291)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 301)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1343)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1625)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1335)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1336)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1337)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1338)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 303)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1339)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1631)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1341)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1342)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 300)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1634)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1629)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1633)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1630)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1612)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1614)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 288)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 289)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1310)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1311)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1315)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1316)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1317)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1318)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1322)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1323)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1326)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 293)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 295)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 296)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1616)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1617)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1619)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1622)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 304)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 305)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1626)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1340)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1627)
I20260808 20:14:34.009907 15408 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:34.034248 12140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1611) took 2973 ms. Trace:
I20260808 20:14:34.038969 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:33.929967 15426 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:34.042135 13014 leader_election.cc:302] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:34.009531 15400 raft_consensus.cc:3037] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:34.050904 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:34.025419 15117 leader_election.cc:288] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:34.024431 15289 leader_election.cc:288] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:33.983023 15096 raft_consensus.cc:3037] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:34.032356 15109 consensus_queue.cc:1046] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20260808 20:14:34.029969 15452 raft_consensus.cc:3032] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:34.032828 15116 raft_consensus.cc:482] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:34.094098 12247 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1839) took 1168 ms. Trace:
I20260808 20:14:34.016995 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:33.997022 11994 leader_election.cc:302] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:34.096735 12206 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:33.968716 15192 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:34.011466 12045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1390) took 1235 ms. Trace:
I20260808 20:14:34.107059 12251 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:33.969937 15333 consensus_queue.cc:260] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:34.103026 12105 service_pool.cc:140] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:53148 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1618)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 291)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 301)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1343)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1625)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1335)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1336)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1337)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1338)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 303)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1339)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1631)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1341)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1342)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 300)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1634)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1629)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1633)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1630)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1612)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1614)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 288)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 289)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1310)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1311)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1315)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1316)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1317)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1318)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1322)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1323)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1326)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 293)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 295)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 296)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1616)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1617)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1619)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1622)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 304)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 305)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1626)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1340)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1627)
I20260808 20:14:33.998667 15311 consensus_queue.cc:260] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:34.046017 15027 consensus_queue.cc:1046] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20260808 20:14:34.048547 15205 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:34.056964 15285 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
W20260808 20:14:34.166617 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 119308us
I20260808 20:14:34.110000 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:34.285305 12001 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:34.285737 12039 raft_consensus.cc:2379] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 4: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:34.279336 15107 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:34.094941 15164 raft_consensus.cc:3037] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:34.291585 12250 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:34.291929 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:34.292583 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:34.292732 12140 rpcz_store.cc:276] 0808 20:14:31.061016 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:31.089549 (+ 28533us) service_pool.cc:227] Handling call
0808 20:14:33.966814 (+2877265us) spinlock_profiling.cc:243] Waited 8.45 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:33.967008 (+ 194us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:34.034003 (+ 66995us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:34.034196 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19425664}
I20260808 20:14:34.293541 12147 raft_consensus.cc:2379] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
W20260808 20:14:34.293955 12147 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 291) took 2734 ms (client timeout 3522 ms). Trace:
W20260808 20:14:34.294286 12147 rpcz_store.cc:269] 0808 20:14:31.559412 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.292377 (+2732965us) service_pool.cc:227] Handling call
0808 20:14:34.293906 (+ 1529us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:34.294370 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:34.295137 12140 raft_consensus.cc:2362] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 8. Re-sending same reply.
I20260808 20:14:34.292565 12039 raft_consensus.cc:3037] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:34.295172 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:34.296312 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:34.111380 12207 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:34.166255 15454 consensus_queue.cc:1046] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.145s
W20260808 20:14:34.301267 15457 log.cc:924] Time spent T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.141s user 0.000s sys 0.001s
I20260808 20:14:34.159430 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:34.304662 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 262594us
W20260808 20:14:34.104563 12141 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1328) took 2495 ms (client timeout 2194 ms). Trace:
I20260808 20:14:34.284658 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:34.305079 12045 rpcz_store.cc:276] 0808 20:14:32.775589 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.010340 (+1234751us) service_pool.cc:227] Handling call
0808 20:14:34.010657 (+ 317us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:34.011229 (+ 572us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:34.011404 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:34.311079 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 199754us
I20260808 20:14:34.311535 12045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1858) took 1392 ms. Trace:
I20260808 20:14:34.311954 12045 rpcz_store.cc:276] 0808 20:14:32.919047 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.310738 (+1391691us) service_pool.cc:227] Handling call
0808 20:14:34.311017 (+ 279us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:34.311176 (+ 159us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:34.311201 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:34.311237 (+ 36us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:34.311267 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:34.311316 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:34.311414 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:34.312991 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:34.306524 12247 rpcz_store.cc:276] 0808 20:14:32.925830 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:32.925937 (+ 107us) service_pool.cc:227] Handling call
0808 20:14:34.093894 (+1167957us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x7b5400374954. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:34.094031 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1061153024}
I20260808 20:14:34.318979 12140 raft_consensus.cc:2362] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 7. Re-sending same reply.
I20260808 20:14:34.319594 12101 leader_election.cc:302] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:34.321945 13016 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:34.323148 13016 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:34.323906 12043 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1379) took 2201 ms. Trace:
I20260808 20:14:34.324281 12043 rpcz_store.cc:276] 0808 20:14:32.122661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:33.872518 (+1749857us) service_pool.cc:227] Handling call
0808 20:14:33.872848 (+ 330us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:33.996287 (+123439us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:33.996313 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:34.003343 (+ 7030us) log.cc:842] Serialized 26 byte log entry
0808 20:14:34.098009 (+ 94666us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:14:34.098042 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:34.098173 (+ 131us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:34.323695 (+225522us) raft_consensus.cc:1671] finished
0808 20:14:34.323714 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:34.323847 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7133,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":203,"thread_start_us":216246,"threads_started":2,"wal-append.queue_time_us":309678}
I20260808 20:14:34.308574 12147 raft_consensus.cc:2379] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 6: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:34.168665 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:34.268553 15352 leader_election.cc:288] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:34.101758 13073 raft_consensus.cc:2454] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:33.987931 12144 rpcz_store.cc:276] 0808 20:14:28.325451 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:29.449958 (+1124507us) service_pool.cc:227] Handling call
0808 20:14:29.450339 (+ 381us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:29.804744 (+354405us) raft_consensus.cc:1520] Early marking committed up to index 4
0808 20:14:30.699744 (+895000us) log.cc:842] Serialized 14 byte log entry
0808 20:14:30.777204 (+ 77460us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:30.793801 (+ 16597us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:30.804626 (+ 10825us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:31.640217 (+835591us) spinlock_profiling.cc:243] Waited 8.57 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:32.900389 (+1260172us) spinlock_profiling.cc:243] Waited 5.44 ms on lock 0x7b18001aad00. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96e2518b1 00007fa96e25e661 00007fa96537ff96 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9
0808 20:14:33.866511 (+966122us) spinlock_profiling.cc:243] Waited 337 ms on lock 0x7b54002a6054. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:33.872406 (+ 5895us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:33.886698 (+ 14292us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5881,"prepare.queue_time_us":694266,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":17209,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":23892,"spinlock_wait_cycles":806667648,"thread_start_us":858979,"threads_started":2,"wal-append.queue_time_us":128008,"wal-append.run_cpu_time_us":14589,"wal-append.run_wall_time_us":1887502}
I20260808 20:14:34.314227 15271 raft_consensus.cc:2781] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election won for term 3
W20260808 20:14:34.301757 15236 consensus_meta.cc:287] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.791s user 0.006s sys 0.003s
I20260808 20:14:34.277809 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:34.297766 15027 raft_consensus.cc:3032] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:34.315227 15432 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:34.315331 15192 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:34.315560 15132 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:34.315708 13072 raft_consensus.cc:3037] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:34.353663 15243 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:34.154028 15104 raft_consensus.cc:3037] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:34.306061 15452 raft_consensus.cc:729] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:34.306468 13014 leader_election.cc:302] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:34.317370 12248 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
W20260808 20:14:34.376773 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 70393.1us
I20260808 20:14:34.377467 13014 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:34.317638 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 321939us
W20260808 20:14:34.379630 12206 leader_election.cc:334] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 2.025s (SENT)
W20260808 20:14:34.380407 12206 leader_election.cc:334] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 2.055s (SENT)
I20260808 20:14:34.318197 12146 raft_consensus.cc:2454] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:34.338677 12047 raft_consensus.cc:2454] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:34.341763 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:34.342204 12043 raft_consensus.cc:1226] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:34.380726 12206 leader_election.cc:302] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:34.283918 15210 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:34.343264 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:34.308832 12039 raft_consensus.cc:2454] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:34.297303 15116 raft_consensus.cc:504] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:34.363891 15470 raft_consensus.cc:3032] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:34.358381 15446 raft_consensus.cc:2781] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 20:14:34.108456 15288 raft_consensus.cc:729] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:34.316329 15322 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:34.310236 12141 rpcz_store.cc:269] 0808 20:14:31.605136 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:33.544553 (+1939417us) service_pool.cc:227] Handling call
0808 20:14:34.100820 (+556267us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:34.566269 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:34.566071 15288 consensus_queue.cc:260] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:34.567129 12141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 303) took 1033 ms. Trace:
I20260808 20:14:34.567248 15288 raft_consensus.cc:3037] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:34.567493 12141 rpcz_store.cc:276] 0808 20:14:33.533882 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.566014 (+1032132us) service_pool.cc:227] Handling call
0808 20:14:34.567068 (+ 1054us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:34.568437 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:34.569150 12141 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:34.570456 15364 leader_election.cc:288] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:34.568624 15204 raft_consensus.cc:504] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:34.354909 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:34.298828 15342 consensus_queue.cc:1159] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" responder_term: 5 status { last_received { term: 4 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260808 20:14:34.111680 13076 raft_consensus.cc:2454] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:34.311136 15109 raft_consensus.cc:3032] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Stepping down as leader of term 6
I20260808 20:14:34.317728 15140 raft_consensus.cc:2781] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:34.539847 15132 raft_consensus.cc:2726] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:34.587378 15140 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:34.587777 15140 raft_consensus.cc:3037] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:34.588164 15204 leader_election.cc:288] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:34.574146 15398 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:34.589239 15398 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:34.329586 15227 raft_consensus.cc:2726] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:34.328925 15198 raft_consensus.cc:2781] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:34.343204 15281 raft_consensus.cc:2726] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:34.368683 12249 raft_consensus.cc:2454] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:34.341374 15408 raft_consensus.cc:2781] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:34.329022 15205 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:34.046001 15234 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:34.329409 12137 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
W20260808 20:14:34.367172 12243 consensus_meta.cc:287] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.747s user 0.006s sys 0.003s
I20260808 20:14:34.358307 13071 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Advancing to term 7
W20260808 20:14:34.381156 12146 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1618) took 2274 ms (client timeout 2692 ms). Trace:
W20260808 20:14:34.606061 12146 rpcz_store.cc:269] 0808 20:14:32.106837 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.284396 (+2177559us) service_pool.cc:227] Handling call
0808 20:14:34.381096 (+ 96700us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:34.607013 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:34.607762 12146 raft_consensus.cc:2354] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7.
I20260808 20:14:34.608196 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1339) took 1060 ms. Trace:
I20260808 20:14:34.608474 12146 rpcz_store.cc:276] 0808 20:14:33.548101 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.606757 (+1058656us) service_pool.cc:227] Handling call
0808 20:14:34.608143 (+ 1386us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:34.517297 12147 raft_consensus.cc:2362] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4. Re-sending same reply.
W20260808 20:14:34.519070 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 141675us
W20260808 20:14:34.518967 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 138566us
I20260808 20:14:34.524314 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:34.360749 15107 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:34.363577 15352 raft_consensus.cc:2726] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:34.371558 15289 raft_consensus.cc:1050] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:14:34.265784 15100 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:34.539552 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 254292us
I20260808 20:14:34.550093 13077 raft_consensus.cc:2454] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:34.168869 15187 consensus_queue.cc:260] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:34.313639 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 316657us
I20260808 20:14:34.365931 15193 leader_election.cc:288] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:34.370232 12241 raft_consensus.cc:3037] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:34.365125 15027 raft_consensus.cc:729] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:34.372825 15271 raft_consensus.cc:686] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:34.284147 12048 rpcz_store.cc:276] 0808 20:14:32.407198 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:33.974580 (+1567382us) service_pool.cc:227] Handling call
0808 20:14:33.989910 (+ 15330us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:34.000919 (+ 11009us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:34.011705 (+ 10786us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:34.013276 (+ 1571us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:34.015026 (+ 1750us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:34.033062 (+ 18036us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:34.051641 (+ 18579us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:34.555577 15458 log.cc:924] Time spent T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.290s user 0.001s sys 0.000s
W20260808 20:14:34.557188 11875 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 257239us
I20260808 20:14:34.372959 15432 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:34.557700 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:34.555507 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:34.565124 15246 raft_consensus.cc:482] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:34.573388 12140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1336) took 1026 ms. Trace:
I20260808 20:14:34.698204 12140 rpcz_store.cc:276] 0808 20:14:33.547278 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.320310 (+773032us) service_pool.cc:227] Handling call
0808 20:14:34.573323 (+253013us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:34.169083 12041 raft_consensus.cc:2454] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:34.575229 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:34.703764 12247 raft_consensus.cc:2396] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260808 20:14:34.586874 15109 raft_consensus.cc:729] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:34.551594 15237 raft_consensus.cc:3037] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:34.540309 15117 raft_consensus.cc:2726] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:34.550200 15470 raft_consensus.cc:729] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:34.516007 15192 consensus_queue.cc:1035] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.228s
I20260808 20:14:34.573303 15334 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:34.712214 15349 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:34.733724 15349 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:34.706144 15246 raft_consensus.cc:504] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:34.717394 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:34.676384 12001 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
W20260808 20:14:34.634141 12137 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1620) took 2502 ms (client timeout 2524 ms). Trace:
I20260808 20:14:34.680617 12102 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:34.778215 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 97675.7us
I20260808 20:14:34.779165 15205 raft_consensus.cc:2781] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 20:14:34.779902 15205 raft_consensus.cc:686] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 7 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:34.537859 15243 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:34.704190 15278 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.012s sys 0.027s
W20260808 20:14:34.781317 15278 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.636s user 0.012s sys 0.027s
I20260808 20:14:34.781493 15205 consensus_queue.cc:237] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:34.782899 15278 raft_consensus.cc:2781] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260808 20:14:34.685324 12206 leader_election.cc:334] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 2.107s (SENT)
W20260808 20:14:34.783921 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 98627.6us
I20260808 20:14:34.684253 12147 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1337) took 1136 ms. Trace:
W20260808 20:14:34.784454 12206 leader_election.cc:334] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 2.125s (SENT)
I20260808 20:14:34.784596 12147 rpcz_store.cc:276] 0808 20:14:33.547547 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.343038 (+795491us) service_pool.cc:227] Handling call
0808 20:14:34.684189 (+341151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:34.784839 12206 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:34.686219 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:34.785691 15117 raft_consensus.cc:2726] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:34.786274 12146 raft_consensus.cc:2454] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:34.624965 15198 raft_consensus.cc:686] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 5 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:34.596084 15446 raft_consensus.cc:686] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:34.743703 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:34.724396 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:34.722949 12144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1338) took 1175 ms. Trace:
I20260808 20:14:34.724853 15271 consensus_queue.cc:237] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:34.723774 15027 consensus_queue.cc:260] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:34.807379 15027 raft_consensus.cc:3037] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
W20260808 20:14:34.808612 12250 consensus_meta.cc:287] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.516s user 0.007s sys 0.009s
I20260808 20:14:34.809016 12250 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:14:34.810281 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:34.810710 15286 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:34.811690 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:34.812376 12250 raft_consensus.cc:2354] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 4. Current term is 5.
I20260808 20:14:34.813721 15446 consensus_queue.cc:237] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:34.814806 13014 leader_election.cc:398] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Vote denied by peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) with higher term. Message: Invalid argument: T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 4. Current term is 5.
I20260808 20:14:34.815155 13014 leader_election.cc:401] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260808 20:14:34.773005 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:34.816536 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:34.817675 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:34.817787 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:34.818291 12247 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
W20260808 20:14:34.818840 12152 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1329) took 3213 ms (client timeout 2249 ms). Trace:
W20260808 20:14:34.718328 15473 log.cc:924] Time spent T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260808 20:14:34.603935 12243 raft_consensus.cc:2454] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:14:34.774611 12045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1859) took 1854 ms. Trace:
I20260808 20:14:34.819121 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:34.825074 15289 consensus_queue.cc:237] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 3, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:34.825778 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:34.827005 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:34.828131 13073 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53290 (request call id 526) took 1122 ms. Trace:
I20260808 20:14:34.828352 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:34.829515 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:34.830608 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:34.831110 12247 raft_consensus.cc:3037] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 4
I20260808 20:14:34.834195 12048 raft_consensus.cc:2396] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260808 20:14:34.835209 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:34.835898 13076 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 662) took 3131 ms (client timeout 2787 ms). Trace:
I20260808 20:14:34.726035 15470 consensus_queue.cc:260] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:34.837598 15470 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:34.823035 12045 rpcz_store.cc:276] 0808 20:14:32.919798 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.316806 (+1397008us) service_pool.cc:227] Handling call
0808 20:14:34.317101 (+ 295us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:34.337389 (+ 20288us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:34.337431 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:34.337475 (+ 44us) raft_consensus.cc:1631] Marking committed up to 1
0808 20:14:34.337508 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:34.774388 (+436880us) spinlock_profiling.cc:243] Waited 20.1 ms on lock 0x7b54004858d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:34.774432 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:34.774556 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":46162688}
I20260808 20:14:34.840246 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:34.836380 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:34.841863 12048 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
W20260808 20:14:34.819206 12152 rpcz_store.cc:269] 0808 20:14:31.605691 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.818650 (+3212959us) service_pool.cc:214] Skipping call since client already timed out
0808 20:14:34.818757 (+ 107us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 20:14:34.782995 15243 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:34.598395 15408 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
W20260808 20:14:34.785889 12206 leader_election.cc:334] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 2.187s (SENT)
I20260808 20:14:34.848197 12206 leader_election.cc:302] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:34.849077 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 63207.8us
I20260808 20:14:34.787180 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:34.778229 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 101957us
W20260808 20:14:34.715524 15460 log.cc:924] Time spent T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.124s user 0.000s sys 0.000s
I20260808 20:14:34.704407 15187 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:14:34.800777 12102 service_pool.cc:137] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:53148 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 2 similar messages]
I20260808 20:14:34.801848 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:34.796028 15278 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:34.805013 15246 leader_election.cc:288] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:34.780082 15204 consensus_queue.cc:1046] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:14:34.844084 12247 raft_consensus.cc:2454] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:34.865684 12247 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:14:34.864742 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:34.866254 12247 raft_consensus.cc:595] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:34.840809 12045 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:34.867796 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:34.866037 12142 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1324) took 3263 ms (client timeout 2055 ms). Trace:
W20260808 20:14:34.868535 12142 rpcz_store.cc:269] 0808 20:14:31.603065 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:32.196618 (+593553us) service_pool.cc:227] Handling call
0808 20:14:34.865874 (+2669256us) spinlock_profiling.cc:243] Waited 833 ms on lock 0x7b5400061154. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:34.865969 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1916881792}
I20260808 20:14:34.846900 12101 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:34.854056 13079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53290 (request call id 517) took 2277 ms. Trace:
I20260808 20:14:34.856348 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:34.869855 11966 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:53148 dropped due to backpressure. The service queue is full; it has 50 items.
I20260808 20:14:34.870242 12146 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:34.869619 13079 rpcz_store.cc:276] 0808 20:14:32.576589 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:32.576701 (+ 112us) service_pool.cc:227] Handling call
0808 20:14:32.954337 (+377636us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:33.069601 (+115264us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:34.853844 (+1784243us) spinlock_profiling.cc:243] Waited 377 ms on lock 0x7b5400581450. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d9a09 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:34.853995 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":867758336}
I20260808 20:14:34.852022 15191 raft_consensus.cc:2726] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:34.871304 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:34.871663 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:34.871922 12146 raft_consensus.cc:2454] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:34.872430 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1629) took 1323 ms. Trace:
I20260808 20:14:34.872702 12146 rpcz_store.cc:276] 0808 20:14:33.549214 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.871064 (+1321850us) service_pool.cc:227] Handling call
0808 20:14:34.872383 (+ 1319us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:34.874063 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:34.857512 12047 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:34.726258 15300 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:14:34.794870 15349 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:34.796396 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
W20260808 20:14:34.777654 12137 rpcz_store.cc:269] 0808 20:14:32.107612 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.095267 (+1987655us) service_pool.cc:227] Handling call
0808 20:14:34.609890 (+514623us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:34.701702 15432 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:34.794229 15426 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:14:34.820600 15400 consensus_meta.cc:287] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.520s user 0.005s sys 0.001s
I20260808 20:14:34.858579 12102 service_pool.cc:140] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:53148 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1629)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1353)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1344)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1349)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1345)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1633)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1350)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1346)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1636)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 310)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 311)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1630)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 308)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1354)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1635)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1347)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1637)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 309)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1612)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1614)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 288)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 289)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1310)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1311)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1315)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1316)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1317)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1318)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1322)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1323)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1326)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 293)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 295)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 296)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1616)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1617)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1619)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1622)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 304)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 305)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1626)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1340)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1627)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1352)
I20260808 20:14:34.854208 15285 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:34.872747 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:34.816689 12105 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:46680 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 300)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1634)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1629)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1353)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1344)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1349)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1345)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1633)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1350)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1346)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1636)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 310)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 311)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1630)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 308)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1354)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1635)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1347)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1637)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 309)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1612)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1614)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 288)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 289)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1310)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1311)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1315)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1316)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1317)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1318)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1322)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1323)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1326)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 293)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 295)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 296)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1616)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1617)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1619)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1622)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 304)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 305)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1626)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1340)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1627)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1352)
W20260808 20:14:34.870635 11966 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:53148 dropped due to backpressure. The service queue is full; it has 50 items.
I20260808 20:14:34.915076 13079 raft_consensus.cc:3037] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:34.877187 12146 raft_consensus.cc:2454] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:34.916756 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:34.917376 12146 raft_consensus.cc:2454] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:34.729573 15096 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:34.730592 15109 consensus_queue.cc:260] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:34.832715 13072 raft_consensus.cc:2454] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:34.801354 12144 rpcz_store.cc:276] 0808 20:14:33.547823 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.367504 (+819681us) service_pool.cc:227] Handling call
0808 20:14:34.722853 (+355349us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:34.922753 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:34.852298 15408 raft_consensus.cc:3037] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:34.824678 15433 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:14:34.836385 13076 rpcz_store.cc:269] 0808 20:14:31.704216 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:31.771261 (+ 67045us) service_pool.cc:227] Handling call
0808 20:14:34.835624 (+3064363us) spinlock_profiling.cc:243] Waited 2.33 s on lock 0x7b5400652054. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:34.835804 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5367715584}
I20260808 20:14:34.908799 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:34.902275 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:34.928960 12142 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:34.929517 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 302) took 1395 ms. Trace:
I20260808 20:14:34.929843 12142 rpcz_store.cc:276] 0808 20:14:33.533617 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.869269 (+1335652us) service_pool.cc:227] Handling call
0808 20:14:34.929459 (+ 60190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:34.918116 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:34.864478 15278 raft_consensus.cc:3037] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
W20260808 20:14:34.822876 12243 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1831) took 2133 ms (client timeout 2170 ms). Trace:
I20260808 20:14:34.929142 12047 raft_consensus.cc:2379] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
W20260808 20:14:34.898026 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 51208us
I20260808 20:14:34.933686 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:34.897737 15243 raft_consensus.cc:3032] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:34.904361 13075 raft_consensus.cc:1261] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:14:34.844198 13073 rpcz_store.cc:276] 0808 20:14:33.705764 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:33.710100 (+ 4336us) service_pool.cc:227] Handling call
0808 20:14:34.827952 (+1117852us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b54004a2c54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:34.828069 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":292689152}
I20260808 20:14:34.898257 15402 raft_consensus.cc:2781] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:34.918057 15206 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:34.941812 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:34.942440 12101 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:34.903065 15269 leader_election.cc:288] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:34.943675 12150 raft_consensus.cc:2379] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 5: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:34.897006 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:34.903460 12138 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1312) took 3579 ms (client timeout 2107 ms). Trace:
I20260808 20:14:34.902695 15234 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:34.947173 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:34.907059 15398 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:34.932809 13079 raft_consensus.cc:2454] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:34.949088 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:34.953337 12247 raft_consensus.cc:2379] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
W20260808 20:14:34.934234 12244 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53030 (request call id 359) took 1381 ms (client timeout 1558 ms). Trace:
W20260808 20:14:34.959800 12244 rpcz_store.cc:269] 0808 20:14:33.553084 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:33.571042 (+ 17958us) service_pool.cc:227] Handling call
0808 20:14:34.922581 (+1351539us) spinlock_profiling.cc:243] Waited 27.3 ms on lock 0x7b58001a1398. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:34.934171 (+ 11590us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":62847232}
I20260808 20:14:34.952556 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:34.909276 15140 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:34.943989 15243 raft_consensus.cc:729] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:34.956036 12101 leader_election.cc:302] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:34.935379 13072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57330 (request call id 442) took 1003 ms. Trace:
I20260808 20:14:34.942777 15109 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:34.931072 15322 raft_consensus.cc:2726] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) with higher term. Message: Invalid argument: T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 4. Current term is 5.
I20260808 20:14:34.963116 12207 leader_election.cc:302] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:34.964161 12000 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:34.990957 12000 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:34.990969 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:34.964978 13079 raft_consensus.cc:1226] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:34.965670 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:34.938172 13071 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
I20260808 20:14:34.928434 15433 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:34.968439 12102 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:34.987416 12247 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:14:34.994547 12247 raft_consensus.cc:595] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:34.938309 12243 rpcz_store.cc:269] 0808 20:14:32.689058 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:32.702053 (+ 12995us) service_pool.cc:227] Handling call
0808 20:14:34.822810 (+2120757us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:34.982296 15164 consensus_meta.cc:287] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.672s user 0.000s sys 0.010s
I20260808 20:14:34.995038 12102 leader_election.cc:302] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:34.992079 13079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 439) took 1082 ms. Trace:
I20260808 20:14:35.001704 13079 rpcz_store.cc:276] 0808 20:14:33.910186 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.963714 (+1053528us) service_pool.cc:227] Handling call
0808 20:14:34.964786 (+ 1072us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:34.991751 (+ 26965us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:34.992026 (+ 275us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:34.997732 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 147587us
I20260808 20:14:34.991717 12047 raft_consensus.cc:2354] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 4. Current term is 5.
I20260808 20:14:34.958081 15204 raft_consensus.cc:2781] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:34.988340 15237 raft_consensus.cc:2726] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260808 20:14:35.004737 12206 leader_election.cc:334] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 2.289s (SENT)
I20260808 20:14:35.005067 13015 leader_election.cc:302] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:35.007539 12001 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:34.988229 15219 leader_election.cc:288] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:34.966989 15402 raft_consensus.cc:686] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:34.980374 13075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53290 (request call id 524) took 2220 ms. Trace:
I20260808 20:14:35.044945 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260808 20:14:35.045013 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:35.051373 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 44942.5us
I20260808 20:14:35.052291 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
W20260808 20:14:35.048887 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 45870.9us
I20260808 20:14:35.052541 13015 leader_election.cc:398] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Vote denied by peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747) with higher term. Message: Invalid argument: T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 4. Current term is 5.
I20260808 20:14:35.052876 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:14:35.053627 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:14:35.053757 13079 raft_consensus.cc:3037] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:35.048501 12206 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:34.932519 15198 consensus_queue.cc:237] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:34.989131 15125 raft_consensus.cc:2781] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:34.982028 12137 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:34.984825 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:35.005213 15289 raft_consensus.cc:2781] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:34.998912 15479 raft_consensus.cc:2781] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:35.079151 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:35.063598 15446 raft_consensus.cc:2726] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:35.008124 15204 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:35.078941 12036 raft_consensus.cc:1226] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 LEADER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:35.074070 12205 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:35.045030 15352 consensus_queue.cc:1046] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.078s
I20260808 20:14:35.115902 15479 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:14:35.128372 12141 consensus_meta.cc:287] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.558s user 0.012s sys 0.000s
I20260808 20:14:35.128513 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:35.128319 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 54354.6us
I20260808 20:14:35.000291 15108 raft_consensus.cc:2781] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:35.081413 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:35.100783 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 96068.6us
I20260808 20:14:35.000262 15477 raft_consensus.cc:2726] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:35.008800 15375 raft_consensus.cc:2726] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:34.984711 13072 rpcz_store.cc:276] 0808 20:14:33.932078 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.016726 (+ 84648us) service_pool.cc:227] Handling call
0808 20:14:34.931751 (+915025us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:35.044775 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:35.148975 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:35.149216 13076 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:35.078110 13079 raft_consensus.cc:2454] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:35.150449 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:35.061728 15337 raft_consensus.cc:2781] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election won for term 7
W20260808 20:14:34.962311 12138 rpcz_store.cc:269] 0808 20:14:31.323903 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:32.053225 (+729322us) service_pool.cc:227] Handling call
0808 20:14:34.774883 (+2721658us) spinlock_profiling.cc:243] Waited 2.85 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:34.903386 (+128503us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6549760}
I20260808 20:14:35.123991 13015 leader_election.cc:302] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:35.072273 15435 consensus_queue.cc:1046] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.053s
I20260808 20:14:35.112206 15289 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:35.129372 13074 raft_consensus.cc:2379] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 4: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:35.008374 15164 raft_consensus.cc:504] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:35.085750 15125 raft_consensus.cc:686] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:35.072865 15219 consensus_queue.cc:1046] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20260808 20:14:35.136610 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:35.171084 12047 raft_consensus.cc:2454] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:35.137576 15140 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:35.138017 15352 raft_consensus.cc:3032] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Stepping down as leader of term 3
W20260808 20:14:35.139413 15104 consensus_meta.cc:287] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.601s user 0.007s sys 0.011s
W20260808 20:14:35.141567 12206 leader_election.cc:334] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 2.402s (SENT)
W20260808 20:14:35.147305 15288 consensus_meta.cc:287] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.580s user 0.008s sys 0.012s
I20260808 20:14:35.131909 12141 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6.
I20260808 20:14:35.147850 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1386) took 2981 ms. Trace:
I20260808 20:14:35.148689 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:35.130119 15204 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:35.116001 15478 raft_consensus.cc:2726] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:35.158197 12001 leader_election.cc:302] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:35.218528 12250 raft_consensus.cc:2454] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
W20260808 20:14:35.218643 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 60518.1us
I20260808 20:14:35.149612 13072 raft_consensus.cc:2454] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
W20260808 20:14:35.180534 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 185154us
I20260808 20:14:35.132017 15479 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:35.134557 15446 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:35.147744 15432 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:35.167734 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 203766us
I20260808 20:14:35.140784 15116 leader_election.cc:288] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:35.148101 14987 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:35.050895 13075 rpcz_store.cc:276] 0808 20:14:32.745819 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:33.529183 (+783364us) service_pool.cc:227] Handling call
0808 20:14:34.903847 (+1374664us) spinlock_profiling.cc:243] Waited 1.47 ms on lock 0x7b5800005198. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:34.904042 (+ 195us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:34.949966 (+ 45924us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:34.960175 (+ 10209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3384192}
I20260808 20:14:35.214092 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1378) took 3095 ms. Trace:
I20260808 20:14:35.153173 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:35.215329 12140 raft_consensus.cc:2379] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:35.217540 12142 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:35.155460 15191 raft_consensus.cc:2781] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election won for term 5
W20260808 20:14:35.161007 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 158622us
I20260808 20:14:35.178623 12046 raft_consensus.cc:1226] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:35.178325 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 1860) took 2258 ms. Trace:
I20260808 20:14:35.219763 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:35.220212 12205 leader_election.cc:302] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:35.214172 12206 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:35.224870 12001 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:35.225293 12048 raft_consensus.cc:2454] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:35.225576 13016 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:35.228367 12254 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:14:35.159570 15289 raft_consensus.cc:3037] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:35.163857 15358 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:35.159224 15337 raft_consensus.cc:686] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
W20260808 20:14:35.246580 13077 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 655) took 4748 ms (client timeout 1679 ms). Trace:
I20260808 20:14:35.241803 12251 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53030 (request call id 360) took 1635 ms. Trace:
I20260808 20:14:35.215711 15108 raft_consensus.cc:686] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
W20260808 20:14:35.239411 12248 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1849) took 1228 ms (client timeout 1628 ms). Trace:
W20260808 20:14:35.242389 12150 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1624) took 2764 ms (client timeout 1641 ms). Trace:
I20260808 20:14:35.129385 15452 consensus_queue.cc:260] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:35.239856 12249 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1838) took 2314 ms. Trace:
I20260808 20:14:35.359274 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:35.306074 12040 rpcz_store.cc:276] 0808 20:14:32.920382 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:33.102069 (+181687us) service_pool.cc:227] Handling call
0808 20:14:35.178163 (+2076094us) spinlock_profiling.cc:243] Waited 459 ms on lock 0x7b540009ef54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:35.178268 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1055220992}
I20260808 20:14:35.243454 12043 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1395) took 1216 ms. Trace:
I20260808 20:14:35.237232 12105 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53162 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1354)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1640)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1635)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1357)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1347)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1364)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1363)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1359)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1361)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1644)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1637)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 309)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1612)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1614)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 288)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 289)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1310)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1311)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1315)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1316)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1317)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1318)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1322)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1323)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1326)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 293)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 295)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 296)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1616)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1617)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1619)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1622)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 304)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 305)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1626)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1340)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1627)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1351)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1352)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1641)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1642)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1355)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1643)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1646)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1356)
I20260808 20:14:35.213479 13074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57330 (request call id 437) took 1337 ms. Trace:
I20260808 20:14:35.307682 12102 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
W20260808 20:14:35.312100 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 85595.6us
I20260808 20:14:35.213593 15352 raft_consensus.cc:729] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:35.320844 12251 rpcz_store.cc:276] 0808 20:14:33.606161 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:33.817826 (+211665us) service_pool.cc:227] Handling call
0808 20:14:35.241643 (+1423817us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b54000bc2d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:35.241744 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":439642496}
W20260808 20:14:35.355613 12248 rpcz_store.cc:269] 0808 20:14:34.011223 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.015738 (+ 4515us) service_pool.cc:227] Handling call
0808 20:14:35.239243 (+1223505us) spinlock_profiling.cc:243] Waited 33.3 ms on lock 0x7b5400086e54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:35.239354 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":76576000}
I20260808 20:14:35.375154 12251 raft_consensus.cc:1261] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260808 20:14:35.313889 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:35.319883 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 95091us
I20260808 20:14:35.306059 12038 rpcz_store.cc:276] 0808 20:14:32.118838 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:33.544166 (+1425328us) service_pool.cc:227] Handling call
0808 20:14:33.572711 (+ 28545us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:34.025024 (+452313us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:34.025052 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:34.025485 (+ 433us) log.cc:842] Serialized 26 byte log entry
0808 20:14:34.142846 (+117361us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:14:34.142884 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:35.169706 (+1026822us) spinlock_profiling.cc:243] Waited 452 ms on lock 0x7b540013e554. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:35.169749 (+ 43us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:35.169794 (+ 45us) raft_consensus.cc:1671] finished
0808 20:14:35.169807 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:35.214017 (+ 44210us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8654,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":10213,"spinlock_wait_cycles":1039418240,"thread_start_us":117036,"threads_started":1,"wal-append.queue_time_us":339864}
I20260808 20:14:35.159009 15237 raft_consensus.cc:2726] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:35.254284 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:35.355785 12150 rpcz_store.cc:269] 0808 20:14:32.478134 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:33.986838 (+1508704us) service_pool.cc:227] Handling call
0808 20:14:35.242191 (+1255353us) spinlock_profiling.cc:243] Waited 851 ms on lock 0x7b5400281ed4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:35.242321 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1956905728}
I20260808 20:14:35.360088 12142 raft_consensus.cc:2454] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
W20260808 20:14:35.321113 13077 rpcz_store.cc:269] 0808 20:14:30.497664 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:30.820040 (+322376us) service_pool.cc:227] Handling call
0808 20:14:35.246367 (+4426327us) spinlock_profiling.cc:243] Waited 3.73 s on lock 0x7b5400670554. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:35.246487 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8576208768}
W20260808 20:14:35.313941 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 172410us
I20260808 20:14:35.427716 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:35.427851 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:35.360176 12043 rpcz_store.cc:276] 0808 20:14:34.026814 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.545033 (+518219us) service_pool.cc:227] Handling call
0808 20:14:34.545345 (+ 312us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:34.692594 (+147249us) raft_consensus.cc:1520] Early marking committed up to index 7
0808 20:14:34.721226 (+ 28632us) log.cc:842] Serialized 14 byte log entry
0808 20:14:34.925631 (+204405us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:34.925676 (+ 45us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:14:34.925719 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:35.243208 (+317489us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b5400170554. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:35.243261 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:35.243382 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":28995,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":2885,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":3444,"spinlock_wait_cycles":338137856,"thread_start_us":203699,"threads_started":1,"wal-append.queue_time_us":274908}
I20260808 20:14:35.224097 15140 raft_consensus.cc:2781] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 20:14:35.245906 15142 raft_consensus.cc:3032] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Stepping down as leader of term 6
I20260808 20:14:35.248466 15333 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:35.359383 12249 rpcz_store.cc:276] 0808 20:14:32.925552 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:32.925678 (+ 126us) service_pool.cc:227] Handling call
0808 20:14:35.239708 (+2314030us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b5400165b54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:35.239802 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3134483200}
W20260808 20:14:35.314076 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 93943.8us
I20260808 20:14:35.357306 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:35.375492 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:35.375703 12251 pending_rounds.cc:76] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a: Aborting all ops after (but not including) 5
I20260808 20:14:35.440488 12251 pending_rounds.cc:97] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260808 20:14:35.440596 12040 raft_consensus.cc:2379] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 4: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:35.172791 12243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 368) took 1129 ms. Trace:
W20260808 20:14:35.376387 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 68796.4us
W20260808 20:14:35.221200 12141 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 306) took 1673 ms (client timeout 2055 ms). Trace:
I20260808 20:14:35.235528 15219 raft_consensus.cc:3032] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Stepping down as leader of term 1
W20260808 20:14:35.247666 12041 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1389) took 2786 ms (client timeout 2893 ms). Trace:
I20260808 20:14:35.311439 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:35.377080 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:35.417399 15375 raft_consensus.cc:2781] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:35.429194 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:35.428326 13077 raft_consensus.cc:3037] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:35.501827 12043 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:14:35.160255 15096 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:35.502825 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:35.503048 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:35.432492 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:35.431608 12245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 362) took 1824 ms. Trace:
I20260808 20:14:35.361231 14987 raft_consensus.cc:2781] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:35.315450 15292 raft_consensus.cc:2781] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:35.162317 15402 consensus_queue.cc:237] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:35.359822 15343 raft_consensus.cc:2781] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:35.332041 15191 raft_consensus.cc:686] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 5 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:35.441365 12251 raft_consensus.cc:2866] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:35.447680 12102 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:35.373740 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:35.593688 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:35.594439 12249 raft_consensus.cc:2354] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6.
I20260808 20:14:35.485874 15098 raft_consensus.cc:482] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:35.378815 12137 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:14:35.460409 13015 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:35.428426 12150 raft_consensus.cc:2454] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:35.416672 15027 raft_consensus.cc:2726] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:35.229017 15446 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:35.321569 15358 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:35.457608 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:35.505193 12245 rpcz_store.cc:276] 0808 20:14:33.606788 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:33.927965 (+321177us) service_pool.cc:227] Handling call
0808 20:14:34.094564 (+166599us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:34.094773 (+ 209us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:34.094799 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:34.094843 (+ 44us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:34.094876 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:34.094934 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:35.431450 (+1336516us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b5400374950. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d9a09 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:35.431556 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":382411008}
I20260808 20:14:35.681725 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:35.682297 12147 raft_consensus.cc:3037] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:35.682334 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:35.683410 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:35.373329 13074 rpcz_store.cc:276] 0808 20:14:33.876229 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:35.128200 (+1251971us) service_pool.cc:227] Handling call
0808 20:14:35.177371 (+ 49171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:35.507237 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:35.435894 15142 raft_consensus.cc:729] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:35.589067 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:35.694108 12047 raft_consensus.cc:2454] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:35.217024 12036 rpcz_store.cc:276] 0808 20:14:32.166717 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:33.893161 (+1726444us) service_pool.cc:227] Handling call
0808 20:14:33.893704 (+ 543us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:35.134170 (+1240466us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:35.147181 (+ 13011us) spinlock_profiling.cc:243] Waited 3.81 ms on lock 0x7b5400491220. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa96826a730 00007fa9682ddb42 00007fa9682d9f26 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772
0808 20:14:35.147632 (+ 451us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b5400170cd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:35.147794 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2572337792}
I20260808 20:14:35.590510 13077 raft_consensus.cc:2454] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:35.506340 15375 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:35.436031 13076 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:35.591442 12251 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 369) took 1550 ms. Trace:
I20260808 20:14:35.697502 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:35.697451 12251 rpcz_store.cc:276] 0808 20:14:34.040654 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:35.374548 (+1333894us) service_pool.cc:227] Handling call
0808 20:14:35.374829 (+ 281us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:35.591192 (+216363us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:35.591385 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:35.501426 15497 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:35.486342 12038 raft_consensus.cc:2379] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 5: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:35.595366 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:35.700620 12142 raft_consensus.cc:2454] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:35.701148 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1635) took 1383 ms. Trace:
I20260808 20:14:35.701434 12142 rpcz_store.cc:276] 0808 20:14:34.317829 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:35.505323 (+1187494us) service_pool.cc:227] Handling call
0808 20:14:35.701093 (+195770us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:35.596242 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 148874us
I20260808 20:14:35.441743 15140 raft_consensus.cc:686] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:35.486864 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:35.488498 15219 raft_consensus.cc:729] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:35.488221 15205 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:35.598691 12206 leader_election.cc:334] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 2.830s (SENT)
I20260808 20:14:35.587049 15343 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:35.603543 15191 consensus_queue.cc:237] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:35.604355 15470 consensus_meta.cc:287] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.766s user 0.011s sys 0.001s
I20260808 20:14:35.604724 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:35.717567 12249 raft_consensus.cc:2454] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:35.717860 12048 raft_consensus.cc:2396] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260808 20:14:35.718461 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:35.719058 12249 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:35.719878 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
W20260808 20:14:35.611985 15493 log.cc:924] Time spent T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.373s user 0.000s sys 0.000s
I20260808 20:14:35.612183 15289 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:35.721305 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:35.721985 13074 raft_consensus.cc:2454] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:35.722154 15289 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:35.722975 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:35.617134 15108 consensus_queue.cc:237] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:35.724421 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:35.508956 15292 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:35.725118 15292 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:35.725507 15219 consensus_queue.cc:260] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:35.726650 15219 raft_consensus.cc:3037] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
W20260808 20:14:35.445405 12141 rpcz_store.cc:269] 0808 20:14:33.547802 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.568199 (+1020397us) service_pool.cc:227] Handling call
0808 20:14:35.219544 (+651345us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:35.489300 15352 consensus_queue.cc:260] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:35.621395 12241 consensus_meta.cc:287] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.925s user 0.011s sys 0.000s
I20260808 20:14:35.621826 15098 raft_consensus.cc:504] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:35.505722 15337 consensus_queue.cc:237] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:35.623755 15164 leader_election.cc:288] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:35.625082 12138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1630) took 1575 ms. Trace:
W20260808 20:14:35.625631 15286 consensus_meta.cc:287] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.815s user 0.007s sys 0.003s
I20260808 20:14:35.629279 13072 raft_consensus.cc:2379] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:35.740666 15098 leader_election.cc:288] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:35.741092 15360 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:35.741245 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:35.741868 13072 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:35.731738 13075 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:35.742815 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:35.631317 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
W20260808 20:14:35.634655 15501 log.cc:924] Time spent T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045: Append to log took a long time: real 0.398s user 0.000s sys 0.001s
I20260808 20:14:35.743535 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:35.629787 15311 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:35.629784 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:35.636744 15187 consensus_meta.cc:287] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.778s user 0.007s sys 0.003s
W20260808 20:14:35.638407 15484 log.cc:924] Time spent T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.409s user 0.001s sys 0.000s
W20260808 20:14:35.639259 15278 consensus_meta.cc:287] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.702s user 0.011s sys 0.000s
I20260808 20:14:35.746135 15278 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:35.748070 15278 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:35.747826 15292 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:35.636188 15109 consensus_meta.cc:287] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.647s user 0.005s sys 0.004s
W20260808 20:14:35.640272 15504 log.cc:924] Time spent T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.405s user 0.000s sys 0.001s
I20260808 20:14:35.749219 13078 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:35.509006 15384 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:35.749743 15292 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:35.750100 12244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1845) took 1792 ms. Trace:
I20260808 20:14:35.749903 15384 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:35.602276 15096 raft_consensus.cc:2781] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:35.750939 15096 raft_consensus.cc:686] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:35.750545 12244 rpcz_store.cc:276] 0808 20:14:33.958092 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.982662 (+1024570us) service_pool.cc:227] Handling call
0808 20:14:34.982998 (+ 336us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:34.983361 (+ 363us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:34.983388 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:34.983828 (+ 440us) log.cc:842] Serialized 26 byte log entry
0808 20:14:35.159319 (+175491us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:14:35.159356 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:35.159418 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:35.639383 (+479965us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:35.749814 (+110431us) raft_consensus.cc:1671] finished
0808 20:14:35.749836 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:35.750042 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37858,"raft.queue_time_us":3335,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":8508,"spinlock_wait_cycles":296931712,"thread_start_us":104176,"threads_started":1,"wal-append.queue_time_us":238531}
I20260808 20:14:35.642004 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:35.642800 15433 consensus_meta.cc:287] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.650s user 0.007s sys 0.002s
I20260808 20:14:35.636695 15435 raft_consensus.cc:979] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a: : Instructing follower 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:14:35.436026 15506 consensus_queue.cc:1035] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.219s
I20260808 20:14:35.427889 15125 consensus_queue.cc:237] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:35.729310 15343 raft_consensus.cc:3037] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:35.754462 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:35.755026 12250 raft_consensus.cc:3037] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:35.446153 12243 rpcz_store.cc:276] 0808 20:14:34.040410 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:35.047838 (+1007428us) service_pool.cc:227] Handling call
0808 20:14:35.115821 (+ 67983us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:35.133662 (+ 17841us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:35.135759 (+ 2097us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:35.145678 (+ 9919us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:35.153815 (+ 8137us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:35.160355 (+ 6540us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:35.169501 (+ 9146us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:35.755884 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:35.757005 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:35.488955 12041 rpcz_store.cc:269] 0808 20:14:32.460787 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:33.069160 (+608373us) service_pool.cc:227] Handling call
0808 20:14:35.247478 (+2178318us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b540004ecd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:35.247585 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":757474560}
I20260808 20:14:35.592530 14987 raft_consensus.cc:482] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:35.712248 15161 consensus_queue.cc:1035] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260808 20:14:35.661093 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:35.608402 15204 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:35.656406 13073 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 436) took 1780 ms. Trace:
I20260808 20:14:35.671545 15432 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:35.673659 12046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1376) took 3568 ms. Trace:
W20260808 20:14:35.771624 12105 connection.cc:232] RPC call timeout handler was delayed by -1.9812477852683514s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20099999999999998s for a response
W20260808 20:14:35.772008 12105 connection.cc:232] RPC call timeout handler was delayed by -1.2819477850571275s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2787s for a response
I20260808 20:14:35.771842 12046 rpcz_store.cc:276] 0808 20:14:32.105240 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:33.542419 (+1437179us) service_pool.cc:227] Handling call
0808 20:14:35.178455 (+1636036us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:35.307404 (+128949us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:35.673470 (+366066us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b540009ef50. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d9a09 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:35.673601 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3686876032}
W20260808 20:14:35.674816 12140 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1341) took 2126 ms (client timeout 2187 ms). Trace:
W20260808 20:14:35.772930 12140 rpcz_store.cc:269] 0808 20:14:33.548617 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.698903 (+1150286us) service_pool.cc:227] Handling call
0808 20:14:35.674634 (+975731us) spinlock_profiling.cc:243] Waited 417 ms on lock 0x7b540018f454. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:35.674759 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":959974144}
I20260808 20:14:35.678659 12101 leader_election.cc:302] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:35.750159 15358 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:35.693535 12043 raft_consensus.cc:2379] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 5: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:35.696710 12001 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:35.698612 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:35.742540 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:35.702061 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:35.781776 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:35.782401 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 85769.9us
I20260808 20:14:35.704846 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:35.705605 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:35.713845 12206 leader_election.cc:302] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:35.784075 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 185407us
I20260808 20:14:35.700721 15375 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:35.733760 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:35.761538 14987 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:14:35.733496 15408 consensus_meta.cc:287] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.792s user 0.004s sys 0.008s
I20260808 20:14:35.741580 15360 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:35.739152 12138 rpcz_store.cc:276] 0808 20:14:34.026170 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:35.163654 (+1137484us) service_pool.cc:227] Handling call
0808 20:14:35.601552 (+437898us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:35.748785 12045 raft_consensus.cc:2354] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3.
I20260808 20:14:35.731890 15205 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:35.782003 15432 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:35.752566 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:35.593014 15403 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:35.818535 12047 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:35.658672 13071 raft_consensus.cc:1226] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:35.819702 13071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 440) took 1903 ms. Trace:
I20260808 20:14:35.820087 13071 rpcz_store.cc:276] 0808 20:14:33.916700 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.992703 (+1076003us) service_pool.cc:227] Handling call
0808 20:14:35.657954 (+665251us) spinlock_profiling.cc:243] Waited 1.29 ms on lock 0x7b5800005198. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:35.658345 (+ 391us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:35.819426 (+161081us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:35.819643 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2976000}
I20260808 20:14:35.760922 13014 leader_election.cc:302] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:35.738622 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:35.743431 13072 raft_consensus.cc:3037] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:35.744257 13075 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:35.768751 12250 raft_consensus.cc:2454] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:35.829375 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:35.753577 15446 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:35.774741 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 96180.4us
I20260808 20:14:35.779649 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:35.699193 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:35.703413 15497 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:35.786630 12043 raft_consensus.cc:2454] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:35.804014 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:35.761798 15435 raft_consensus.cc:1067] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Signalling peer 4cd46f13bceb4c3c9ef71e755e17b995 to start an election
I20260808 20:14:35.735690 12241 raft_consensus.cc:2454] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:35.789395 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:35.752697 15479 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:35.855616 15384 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:35.840157 15096 consensus_queue.cc:237] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:35.856666 15479 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:35.857777 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:35.795128 12206 leader_election.cc:334] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.989s (SENT)
I20260808 20:14:35.859184 12206 leader_election.cc:302] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:35.859987 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 64961.1us
W20260808 20:14:35.861914 12206 leader_election.cc:334] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 2.194s (SENT)
I20260808 20:14:35.862103 15497 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:35.862318 12206 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:35.862991 15476 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:35.863461 15476 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:35.863981 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:35.864542 12146 raft_consensus.cc:2454] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:35.865337 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:35.865886 12146 raft_consensus.cc:2454] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:35.692464 12147 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:14:35.771649 13073 rpcz_store.cc:276] 0808 20:14:33.875945 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.966297 (+1090352us) service_pool.cc:227] Handling call
0808 20:14:35.655185 (+688888us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b5800005198. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537abce 000000000034aea9 00007fa96d1a6a30 00007fa96d26c7b4 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:35.655440 (+ 255us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:35.655744 (+ 304us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:35.655808 (+ 64us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:35.655911 (+ 103us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:35.655983 (+ 72us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:35.656106 (+ 123us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:35.656313 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23051136}
I20260808 20:14:35.867861 13073 raft_consensus.cc:1226] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
W20260808 20:14:35.868093 12147 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1342) took 2319 ms (client timeout 2231 ms). Trace:
W20260808 20:14:35.868417 12147 rpcz_store.cc:269] 0808 20:14:33.548840 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.785331 (+1236491us) service_pool.cc:227] Handling call
0808 20:14:35.681585 (+896254us) spinlock_profiling.cc:243] Waited 2.37 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:35.868026 (+186441us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5454080}
W20260808 20:14:35.854945 12241 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53030 (request call id 367) took 1814 ms (client timeout 1529 ms). Trace:
I20260808 20:14:35.774554 15164 raft_consensus.cc:482] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:35.759676 12150 raft_consensus.cc:2454] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:35.871826 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:35.872560 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:35.781136 15477 raft_consensus.cc:2781] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:35.873162 12150 raft_consensus.cc:2454] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:35.873610 12150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1637) took 1073 ms. Trace:
I20260808 20:14:35.873878 12150 rpcz_store.cc:276] 0808 20:14:34.800600 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:35.872352 (+1071752us) service_pool.cc:227] Handling call
0808 20:14:35.873566 (+ 1214us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:35.793162 15098 raft_consensus.cc:2726] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 20:14:35.788539 15358 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:35.794158 15506 raft_consensus.cc:2781] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:35.735981 15142 consensus_queue.cc:260] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:35.793679 15286 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:35.761734 15433 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:35.869426 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:35.840305 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:35.886932 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:35.818529 15205 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:35.827363 15161 raft_consensus.cc:2726] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260808 20:14:35.822003 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 61185us
I20260808 20:14:35.844960 13071 raft_consensus.cc:1226] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:35.814316 15408 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:35.779112 15140 consensus_queue.cc:237] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:35.836091 12245 raft_consensus.cc:2454] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:35.874725 15477 raft_consensus.cc:482] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:35.915609 15477 raft_consensus.cc:3037] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:35.915975 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:35.917147 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:35.918167 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:35.919225 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:35.919763 12245 raft_consensus.cc:3037] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
W20260808 20:14:35.920461 15507 log.cc:924] Time spent T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.269s user 0.001s sys 0.001s
I20260808 20:14:35.871222 15198 raft_consensus.cc:2726] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:35.871385 15435 raft_consensus.cc:2726] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:35.866583 15476 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:35.879966 13074 raft_consensus.cc:1226] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7. Ops: []
I20260808 20:14:35.898646 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:35.925348 12251 raft_consensus.cc:2396] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260808 20:14:35.899262 15142 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:35.909338 13071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 447) took 1874 ms. Trace:
I20260808 20:14:35.887323 15286 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:35.932217 15408 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:35.885075 15506 raft_consensus.cc:686] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:35.933362 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:35.877161 15164 raft_consensus.cc:504] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:35.882567 15358 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:35.967079 15433 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:35.971592 15286 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:35.973172 15164 leader_election.cc:288] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:35.965356 15343 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:35.981256 15506 consensus_queue.cc:237] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:35.982990 15343 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:35.962222 15204 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:35.984930 15477 raft_consensus.cc:504] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:35.985443 15375 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:35.986967 15477 leader_election.cc:288] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:35.987648 15375 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:35.988878 15426 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.751s user 0.017s sys 0.029s
I20260808 20:14:35.791730 11994 leader_election.cc:398] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Vote denied by peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) with higher term. Message: Invalid argument: T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6.
I20260808 20:14:35.815799 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:35.826592 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:35.849213 12140 raft_consensus.cc:2454] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:35.786753 12038 raft_consensus.cc:2396] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260808 20:14:35.869338 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:35.871062 12241 rpcz_store.cc:269] 0808 20:14:34.040144 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.242319 (+202175us) service_pool.cc:227] Handling call
0808 20:14:35.854864 (+1612545us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:35.837791 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:35.848264 12152 raft_consensus.cc:2454] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:14:35.866575 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:35.838145 12101 leader_election.cc:302] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:35.822116 12105 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:35.989748 15426 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.751s user 0.018s sys 0.029s
I20260808 20:14:36.037520 12250 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:35.911685 12248 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1851) took 1811 ms. Trace:
I20260808 20:14:35.957665 13073 raft_consensus.cc:3037] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:36.040421 15477 raft_consensus.cc:2781] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260808 20:14:35.934556 12254 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53030 (request call id 352) took 4497 ms (client timeout 2210 ms). Trace:
I20260808 20:14:36.041025 15477 raft_consensus.cc:482] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:36.041456 15477 raft_consensus.cc:3037] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
W20260808 20:14:36.041339 12254 rpcz_store.cc:269] 0808 20:14:31.436838 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:31.483986 (+ 47148us) service_pool.cc:227] Handling call
0808 20:14:33.854899 (+2370913us) spinlock_profiling.cc:243] Waited 3.59 ms on lock 0x7b58001a1398. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:35.934375 (+2079476us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54000ef954. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:35.934490 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2884366464}
I20260808 20:14:36.018922 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:36.044456 12036 raft_consensus.cc:2379] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 3: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:35.959913 13074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 443) took 2025 ms. Trace:
I20260808 20:14:36.045679 13074 rpcz_store.cc:276] 0808 20:14:33.934605 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:35.798612 (+1864007us) service_pool.cc:227] Handling call
0808 20:14:35.854724 (+ 56112us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:35.952802 (+ 98078us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:35.958961 (+ 6159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.020349 12041 raft_consensus.cc:2379] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 5: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:36.047927 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
W20260808 20:14:36.019843 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 228071us
I20260808 20:14:35.833668 12144 raft_consensus.cc:2354] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for earlier term 5. Current term is 6.
W20260808 20:14:36.032239 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 194004us
I20260808 20:14:36.048144 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49226 (request call id 452) took 1114 ms. Trace:
I20260808 20:14:36.027748 12147 raft_consensus.cc:2379] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
W20260808 20:14:36.029326 12152 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 300) took 2496 ms (client timeout 2443 ms). Trace:
W20260808 20:14:36.034731 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 212551us
I20260808 20:14:36.035331 12243 raft_consensus.cc:2454] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:36.035672 12247 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1847) took 2075 ms. Trace:
I20260808 20:14:36.022720 12138 raft_consensus.cc:2379] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 8: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
W20260808 20:14:36.049176 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 588651us
I20260808 20:14:36.029889 12146 raft_consensus.cc:2379] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:36.058250 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:36.059128 13075 raft_consensus.cc:2454] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:36.040371 12248 rpcz_store.cc:276] 0808 20:14:34.100611 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:35.416638 (+1316027us) service_pool.cc:227] Handling call
0808 20:14:35.439844 (+ 23206us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:35.457937 (+ 18093us) raft_consensus.cc:1520] Early marking committed up to index 2
0808 20:14:35.457982 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:35.458440 (+ 458us) log.cc:842] Serialized 26 byte log entry
0808 20:14:35.789514 (+331074us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:14:35.789550 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:35.789610 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:35.911455 (+121845us) raft_consensus.cc:1671] finished
0808 20:14:35.911478 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:35.911624 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":79,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":56101,"thread_start_us":196505,"threads_started":1,"wal-append.queue_time_us":346147}
I20260808 20:14:36.033470 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
W20260808 20:14:36.055850 12152 rpcz_store.cc:269] 0808 20:14:33.533039 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:35.244698 (+1711659us) service_pool.cc:227] Handling call
0808 20:14:35.660920 (+416222us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:36.029062 (+368142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":42637696}
I20260808 20:14:36.060814 15506 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:36.022253 12039 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:36.026597 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:36.055442 12041 rpcz_store.cc:276] 0808 20:14:34.933257 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:35.766815 (+833558us) service_pool.cc:227] Handling call
0808 20:14:36.048046 (+281231us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.056435 12247 rpcz_store.cc:276] 0808 20:14:33.960098 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.998490 (+1038392us) service_pool.cc:227] Handling call
0808 20:14:35.085683 (+ 87193us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:35.100144 (+ 14461us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:35.100176 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:35.114142 (+ 13966us) log.cc:842] Serialized 26 byte log entry
0808 20:14:35.379715 (+265573us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:35.379757 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:35.379828 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:35.919879 (+540051us) spinlock_profiling.cc:243] Waited 36.4 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:36.034922 (+115043us) raft_consensus.cc:1671] finished
0808 20:14:36.034972 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:36.035443 (+ 471us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34879,"raft.queue_time_us":27034,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":10450,"spinlock_wait_cycles":83639552,"thread_start_us":219609,"threads_started":1,"wal-append.queue_time_us":364341}
I20260808 20:14:36.033308 12205 leader_election.cc:302] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:36.028053 12043 raft_consensus.cc:2454] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:36.042999 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:36.026855 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:35.952525 13071 rpcz_store.cc:276] 0808 20:14:34.034640 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:35.821088 (+1786448us) service_pool.cc:227] Handling call
0808 20:14:35.825681 (+ 4593us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:35.909017 (+ 83336us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:35.909268 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:36.082446 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 378383us
I20260808 20:14:36.083766 13071 raft_consensus.cc:1226] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:36.084604 13071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53290 (request call id 537) took 1018 ms. Trace:
I20260808 20:14:36.071275 12048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49226 (request call id 447) took 1194 ms. Trace:
I20260808 20:14:36.096148 12048 rpcz_store.cc:276] 0808 20:14:34.876296 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:35.312646 (+436350us) service_pool.cc:227] Handling call
0808 20:14:36.070864 (+758218us) spinlock_profiling.cc:243] Waited 342 ms on lock 0x7b540034ec54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:36.071096 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":785505152}
I20260808 20:14:36.077436 15446 raft_consensus.cc:2726] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:36.099057 13014 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:36.099695 13080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53290 (request call id 518) took 3522 ms. Trace:
I20260808 20:14:36.100526 15446 raft_consensus.cc:2781] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:36.101135 15446 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:36.101220 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 1872) took 1137 ms. Trace:
I20260808 20:14:36.095498 13071 rpcz_store.cc:276] 0808 20:14:35.065773 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.082736 (+1016963us) service_pool.cc:227] Handling call
0808 20:14:36.083276 (+ 540us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.084304 (+ 1028us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.084525 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.102191 12038 rpcz_store.cc:276] 0808 20:14:34.963507 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.026199 (+1062692us) service_pool.cc:227] Handling call
0808 20:14:36.101126 (+ 74927us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.103178 12048 raft_consensus.cc:1226] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 5. Ops: []
W20260808 20:14:36.069763 12144 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1350) took 1743 ms (client timeout 2289 ms). Trace:
W20260808 20:14:36.020529 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 222194us
I20260808 20:14:36.023733 12141 raft_consensus.cc:2379] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 4: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:36.096611 12254 raft_consensus.cc:2362] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6. Re-sending same reply.
I20260808 20:14:36.021535 12046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1404) took 1014 ms. Trace:
I20260808 20:14:36.106091 12046 rpcz_store.cc:276] 0808 20:14:35.006676 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:35.779737 (+773061us) service_pool.cc:227] Handling call
0808 20:14:36.021261 (+241524us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.106808 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:36.107681 12254 raft_consensus.cc:2454] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:36.102815 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:36.071162 12105 leader_election.cc:302] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:36.079399 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:36.079986 12147 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1644) took 1121 ms. Trace:
I20260808 20:14:36.108824 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:36.079614 15506 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:36.081199 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:36.081444 12101 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:36.081940 12140 raft_consensus.cc:2454] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:36.085043 12146 raft_consensus.cc:1226] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 7. Current term is 8. Ops: []
I20260808 20:14:36.111352 12241 raft_consensus.cc:3037] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:36.113256 12243 raft_consensus.cc:2379] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:36.113620 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:36.114311 12047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49226 (request call id 443) took 1256 ms. Trace:
I20260808 20:14:36.114465 12140 raft_consensus.cc:2454] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:36.115180 12243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 372) took 1417 ms. Trace:
I20260808 20:14:36.086911 15358 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:36.090797 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:36.028574 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:36.092588 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:36.093130 12043 raft_consensus.cc:1226] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:36.118844 12043 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1870) took 1325 ms. Trace:
I20260808 20:14:36.119212 12043 rpcz_store.cc:276] 0808 20:14:34.793608 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.092366 (+1298758us) service_pool.cc:227] Handling call
0808 20:14:36.092823 (+ 457us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.118583 (+ 25760us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.118771 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.119905 15477 raft_consensus.cc:504] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:36.094875 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1863) took 1569 ms. Trace:
I20260808 20:14:36.120446 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:36.090015 15452 raft_consensus.cc:3037] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:36.048843 13074 raft_consensus.cc:3037] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:36.101639 15446 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:36.072111 11994 leader_election.cc:302] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:36.078156 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1862) took 1530 ms. Trace:
I20260808 20:14:36.110235 12048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 441) took 1254 ms. Trace:
I20260808 20:14:36.084622 12138 raft_consensus.cc:1261] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 20:14:36.115939 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:36.101728 15352 raft_consensus.cc:3037] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:14:36.104957 12144 rpcz_store.cc:269] 0808 20:14:34.326424 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.942919 (+616495us) service_pool.cc:227] Handling call
0808 20:14:36.069344 (+1126425us) spinlock_profiling.cc:243] Waited 702 ms on lock 0x7b54001801d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:36.069570 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1614822656}
I20260808 20:14:36.139737 12144 raft_consensus.cc:1226] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:36.140424 12144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 289) took 4885 ms. Trace:
I20260808 20:14:36.120319 12150 raft_consensus.cc:2454] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:36.094406 15243 consensus_queue.cc:260] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:36.142050 15243 raft_consensus.cc:3037] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:36.145457 15358 raft_consensus.cc:482] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:36.146240 15358 raft_consensus.cc:504] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:36.147331 15432 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:36.148344 15360 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:36.148615 12142 raft_consensus.cc:2379] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 6: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:36.149130 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1358) took 1016 ms. Trace:
I20260808 20:14:36.149420 12142 rpcz_store.cc:276] 0808 20:14:35.133051 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.119939 (+986888us) service_pool.cc:227] Handling call
0808 20:14:36.149061 (+ 29122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.150049 15360 consensus_queue.cc:1046] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:36.150964 12142 raft_consensus.cc:1226] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260808 20:14:36.151631 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1310) took 4828 ms. Trace:
I20260808 20:14:36.151942 12142 rpcz_store.cc:276] 0808 20:14:31.323171 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.150097 (+4826926us) service_pool.cc:227] Handling call
0808 20:14:36.150593 (+ 496us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:36.151382 (+ 789us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.151579 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.153132 15142 raft_consensus.cc:3032] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:36.141259 12150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 309) took 1352 ms. Trace:
I20260808 20:14:36.141325 12254 raft_consensus.cc:2396] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 20:14:36.120591 12043 raft_consensus.cc:1226] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3. Ops: []
W20260808 20:14:36.120994 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 88138.9us
I20260808 20:14:36.120785 12041 rpcz_store.cc:276] 0808 20:14:34.525413 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.089168 (+1563755us) service_pool.cc:227] Handling call
0808 20:14:36.089726 (+ 558us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.090111 (+ 385us) raft_consensus.cc:1520] Early marking committed up to index 2
0808 20:14:36.090153 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:36.094536 (+ 4383us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:14:36.094583 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.094655 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:36.094782 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.123080 15506 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:36.124493 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 53487.1us
I20260808 20:14:36.100147 13080 rpcz_store.cc:276] 0808 20:14:32.577610 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:33.516215 (+938605us) service_pool.cc:227] Handling call
0808 20:14:35.247019 (+1730804us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:35.249281 (+ 2262us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:35.249311 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:35.249355 (+ 44us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:35.249385 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:35.249444 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:36.099297 (+849853us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b5400670550. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d9a09 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:36.099471 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1921,"raft.queue_time_us":2358,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":160,"spinlock_wait_cycles":3903296896}
I20260808 20:14:36.125730 12045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49226 (request call id 442) took 1273 ms. Trace:
I20260808 20:14:36.127065 15342 consensus_queue.cc:1046] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 20:14:36.127797 15245 raft_consensus.cc:2726] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:36.109529 12147 rpcz_store.cc:276] 0808 20:14:34.958827 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:35.869126 (+910299us) service_pool.cc:227] Handling call
0808 20:14:36.079876 (+210750us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.166597 12147 raft_consensus.cc:1226] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 1. Current term is 3. Ops: []
I20260808 20:14:36.167189 12147 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1315) took 4704 ms. Trace:
I20260808 20:14:36.167457 12147 rpcz_store.cc:276] 0808 20:14:31.463204 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.166038 (+4702834us) service_pool.cc:227] Handling call
0808 20:14:36.166447 (+ 409us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.166966 (+ 519us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.167142 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.155802 15142 raft_consensus.cc:729] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:36.148252 15358 leader_election.cc:288] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:36.169492 15100 raft_consensus.cc:2781] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260808 20:14:36.169682 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 88373.8us
I20260808 20:14:36.170045 15100 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:36.170526 15100 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:36.173652 13015 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:36.174481 12105 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:36.174993 14987 raft_consensus.cc:504] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:36.175534 13015 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:36.176661 15204 raft_consensus.cc:2781] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:36.140748 12144 rpcz_store.cc:276] 0808 20:14:31.255013 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.138982 (+4883969us) service_pool.cc:227] Handling call
0808 20:14:36.139444 (+ 462us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.140149 (+ 705us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.140328 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.178617 12144 raft_consensus.cc:1226] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:36.180181 15477 leader_election.cc:288] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:36.169520 15142 consensus_queue.cc:260] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:36.181206 15142 raft_consensus.cc:3037] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:36.112808 12251 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1852) took 1523 ms. Trace:
I20260808 20:14:36.193022 15506 raft_consensus.cc:482] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:36.193259 12251 rpcz_store.cc:276] 0808 20:14:34.588992 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.111636 (+1522644us) service_pool.cc:227] Handling call
0808 20:14:36.112107 (+ 471us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.112460 (+ 353us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:36.112484 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:36.112537 (+ 53us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:36.112567 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.112629 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:36.112737 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.193746 15506 raft_consensus.cc:504] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:36.197608 15506 leader_election.cc:288] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:36.199090 12251 raft_consensus.cc:1226] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:36.201685 12041 raft_consensus.cc:1261] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 20:14:36.202462 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1873) took 1236 ms. Trace:
I20260808 20:14:36.202867 12041 rpcz_store.cc:276] 0808 20:14:34.966047 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.200698 (+1234651us) service_pool.cc:227] Handling call
0808 20:14:36.201110 (+ 412us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.202126 (+ 1016us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.202368 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.077436 15375 raft_consensus.cc:3037] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:36.205263 12144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1317) took 4709 ms. Trace:
I20260808 20:14:36.136528 12048 rpcz_store.cc:276] 0808 20:14:34.851097 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.102463 (+1251366us) service_pool.cc:227] Handling call
0808 20:14:36.102889 (+ 426us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.103612 (+ 723us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.105260 (+ 1648us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.209111 12138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1614) took 5145 ms. Trace:
I20260808 20:14:36.209981 12048 raft_consensus.cc:1226] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 1. Current term is 2. Ops: []
I20260808 20:14:36.210614 12048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1400) took 1204 ms. Trace:
I20260808 20:14:36.210927 12048 rpcz_store.cc:276] 0808 20:14:35.006072 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.209259 (+1203187us) service_pool.cc:227] Handling call
0808 20:14:36.209685 (+ 426us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.210387 (+ 702us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.210555 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.212381 12048 raft_consensus.cc:1226] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260808 20:14:36.114705 12047 rpcz_store.cc:276] 0808 20:14:34.857687 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:35.694572 (+836885us) service_pool.cc:227] Handling call
0808 20:14:36.113909 (+419337us) spinlock_profiling.cc:243] Waited 36.1 ms on lock 0x7b5400007b54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:36.114113 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":82942720}
I20260808 20:14:36.213788 12047 raft_consensus.cc:1226] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:36.214340 12047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1402) took 1207 ms. Trace:
I20260808 20:14:36.214682 12047 rpcz_store.cc:276] 0808 20:14:35.006371 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.213224 (+1206853us) service_pool.cc:227] Handling call
0808 20:14:36.213589 (+ 365us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.214122 (+ 533us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.214282 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.215369 12141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1347) took 1917 ms. Trace:
I20260808 20:14:36.215775 12141 rpcz_store.cc:276] 0808 20:14:34.298342 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:35.734372 (+1436030us) service_pool.cc:227] Handling call
0808 20:14:36.215123 (+480751us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b540030d754. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:36.215290 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":449230720}
I20260808 20:14:36.216755 13072 raft_consensus.cc:2454] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:36.154459 12150 rpcz_store.cc:276] 0808 20:14:34.788833 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:35.874464 (+1085631us) service_pool.cc:227] Handling call
0808 20:14:36.141162 (+266698us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.121034 15505 raft_consensus.cc:482] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:36.218062 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1612) took 5156 ms. Trace:
I20260808 20:14:36.218587 15505 raft_consensus.cc:504] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:36.219703 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 1861) took 1698 ms. Trace:
I20260808 20:14:36.220075 12039 rpcz_store.cc:276] 0808 20:14:34.521401 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.524025 (+ 2624us) service_pool.cc:227] Handling call
0808 20:14:36.219514 (+1695489us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54003d0254. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:36.219650 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3098696960}
I20260808 20:14:36.218427 12146 rpcz_store.cc:276] 0808 20:14:31.061254 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.067034 (+5005780us) service_pool.cc:227] Handling call
0808 20:14:36.084767 (+ 17733us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.111761 (+ 26994us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.217844 (+106083us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x7b54002a7450. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d9a09 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:36.217987 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39507200}
I20260808 20:14:36.222245 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:36.222986 12146 raft_consensus.cc:2362] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2. Re-sending same reply.
I20260808 20:14:36.223990 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:36.220793 15505 leader_election.cc:288] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:36.160380 12043 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 455) took 1199 ms. Trace:
I20260808 20:14:36.225989 12043 rpcz_store.cc:276] 0808 20:14:34.957828 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.120096 (+1162268us) service_pool.cc:227] Handling call
0808 20:14:36.120434 (+ 338us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.156830 (+ 36396us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.157012 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.226600 12250 raft_consensus.cc:1261] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:14:36.227629 12043 raft_consensus.cc:1226] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:36.161698 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:36.180629 15100 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
W20260808 20:14:36.229462 13078 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53290 (request call id 528) took 2188 ms (client timeout 1974 ms). Trace:
I20260808 20:14:36.229990 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:36.230657 15100 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:36.231400 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:36.181510 15116 raft_consensus.cc:2726] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:36.232214 13071 raft_consensus.cc:3037] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:36.182140 15332 raft_consensus.cc:2781] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:36.232752 15332 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:36.182734 15342 consensus_queue.cc:1225] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer e5bd9a4616ce4acfa20dc7a002984a20 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260808 20:14:36.229851 13078 rpcz_store.cc:269] 0808 20:14:34.041377 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.817545 (+776168us) service_pool.cc:227] Handling call
0808 20:14:36.229099 (+1411554us) spinlock_profiling.cc:243] Waited 931 ms on lock 0x7b5400710f54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:36.229260 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2140657408}
I20260808 20:14:36.233186 15332 raft_consensus.cc:3037] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:36.234205 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:36.235353 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:36.236203 13078 raft_consensus.cc:2396] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260808 20:14:36.167510 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:36.158810 12045 rpcz_store.cc:276] 0808 20:14:34.852203 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:34.867563 (+ 15360us) service_pool.cc:227] Handling call
0808 20:14:36.115890 (+1248327us) spinlock_profiling.cc:243] Waited 880 ms on lock 0x7b540009ea54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:36.125474 (+ 9584us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2024471296}
I20260808 20:14:36.230666 12248 raft_consensus.cc:1261] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 20:14:36.225096 12146 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:36.238104 15426 raft_consensus.cc:2741] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260808 20:14:36.240912 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:36.241178 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:36.242040 13078 raft_consensus.cc:2379] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 4: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:36.242215 15332 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:36.094159 12206 leader_election.cc:334] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 2.249s (SENT)
I20260808 20:14:36.187222 15204 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:36.188995 12137 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:36.191473 12142 raft_consensus.cc:1226] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 7. Current term is 8. Ops: [6.8-7.9]
I20260808 20:14:36.153262 12038 raft_consensus.cc:1226] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 5. Ops: []
I20260808 20:14:36.217998 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:36.186800 12245 raft_consensus.cc:2454] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:36.168848 12147 raft_consensus.cc:1226] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:36.242290 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:36.244132 12048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1401) took 1237 ms. Trace:
I20260808 20:14:36.259462 12048 rpcz_store.cc:276] 0808 20:14:35.006222 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.211658 (+1205436us) service_pool.cc:227] Handling call
0808 20:14:36.212113 (+ 455us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:36.212716 (+ 603us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.244076 (+ 31360us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.227630 12101 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:36.251221 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1311) took 4927 ms. Trace:
I20260808 20:14:36.260499 12142 rpcz_store.cc:276] 0808 20:14:31.323516 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.152811 (+4829295us) service_pool.cc:227] Handling call
0808 20:14:36.191153 (+ 38342us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 20:14:36.248396 (+ 57243us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.251161 (+ 2765us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.261262 12045 raft_consensus.cc:2454] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:36.262048 13016 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:36.205724 12144 rpcz_store.cc:276] 0808 20:14:31.495885 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.177805 (+4681920us) service_pool.cc:227] Handling call
0808 20:14:36.178288 (+ 483us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.179083 (+ 795us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.205171 (+ 26088us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.251812 13014 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:36.254962 13072 raft_consensus.cc:1226] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:36.209600 12138 rpcz_store.cc:276] 0808 20:14:31.063505 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.083561 (+5020056us) service_pool.cc:227] Handling call
0808 20:14:36.084092 (+ 531us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.138095 (+ 54003us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.209021 (+ 70926us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.256140 12244 raft_consensus.cc:2454] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:36.272258 12147 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1316) took 4786 ms. Trace:
I20260808 20:14:36.272599 12147 rpcz_store.cc:276] 0808 20:14:31.486004 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.168178 (+4682174us) service_pool.cc:227] Handling call
0808 20:14:36.168558 (+ 380us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.269093 (+100535us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.272203 (+ 3110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.115633 12243 rpcz_store.cc:276] 0808 20:14:34.697800 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.113995 (+1416195us) service_pool.cc:227] Handling call
0808 20:14:36.114472 (+ 477us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.114792 (+ 320us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:36.114826 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:36.114900 (+ 74us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:36.114940 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.114999 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:36.115114 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.262521 12137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1364) took 1025 ms. Trace:
I20260808 20:14:36.276373 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 446) took 1389 ms. Trace:
I20260808 20:14:36.276799 12038 rpcz_store.cc:276] 0808 20:14:34.875956 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.103108 (+1227152us) service_pool.cc:227] Handling call
0808 20:14:36.104934 (+ 1826us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.264803 (+159869us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.265005 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.262905 12150 raft_consensus.cc:2362] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6. Re-sending same reply.
I20260808 20:14:36.252877 15332 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:36.263866 12245 raft_consensus.cc:1226] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:36.258704 12206 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:36.264201 12248 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1854) took 1583 ms. Trace:
I20260808 20:14:36.262212 12142 raft_consensus.cc:1261] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260808 20:14:36.275595 13072 raft_consensus.cc:1261] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260808 20:14:36.275787 15192 raft_consensus.cc:3032] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:36.263172 15116 raft_consensus.cc:2781] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:36.280969 12248 rpcz_store.cc:276] 0808 20:14:34.681239 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.111888 (+1430649us) service_pool.cc:227] Handling call
0808 20:14:36.230278 (+118390us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.237542 (+ 7264us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.263999 (+ 26457us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b5400374e50. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d9a09 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:36.264131 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":271328000}
I20260808 20:14:36.248061 15204 raft_consensus.cc:3037] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:36.283033 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1322) took 4766 ms. Trace:
I20260808 20:14:36.293547 12142 rpcz_store.cc:276] 0808 20:14:31.516767 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.261346 (+4744579us) service_pool.cc:227] Handling call
0808 20:14:36.261780 (+ 434us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.281038 (+ 19258us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.282970 (+ 1932us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.262732 12045 raft_consensus.cc:1226] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 LEADER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7. Ops: []
I20260808 20:14:36.137507 12036 rpcz_store.cc:276] 0808 20:14:34.525148 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.045332 (+1520184us) service_pool.cc:227] Handling call
0808 20:14:36.045975 (+ 643us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.046530 (+ 555us) raft_consensus.cc:1520] Early marking committed up to index 3
0808 20:14:36.046597 (+ 67us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:36.046667 (+ 70us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:36.046711 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.046799 (+ 88us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:36.055608 (+ 8809us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.261529 15278 raft_consensus.cc:2781] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:36.231894 15433 raft_consensus.cc:2932] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:36.272183 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:36.284695 13016 leader_election.cc:334] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 2.180s (SENT)
I20260808 20:14:36.292625 13078 raft_consensus.cc:1261] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260808 20:14:36.281913 13072 pending_rounds.cc:76] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20: Aborting all ops after (but not including) 5
W20260808 20:14:36.297796 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 46117.5us
I20260808 20:14:36.283396 15192 raft_consensus.cc:729] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:36.341413 12036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b"
dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:53018
I20260808 20:14:36.346066 12000 leader_election.cc:302] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:36.346379 12241 raft_consensus.cc:2454] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:36.349713 13077 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 448) took 2037 ms. Trace:
I20260808 20:14:36.344591 12048 raft_consensus.cc:1226] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 1. Current term is 2. Ops: []
W20260808 20:14:36.343695 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 116165us
I20260808 20:14:36.343274 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 296) took 4520 ms. Trace:
I20260808 20:14:36.344969 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:36.370963 12251 raft_consensus.cc:1226] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260808 20:14:36.353504 13074 raft_consensus.cc:2454] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:36.298768 15278 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:36.336539 12254 raft_consensus.cc:1261] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 20:14:36.359076 12146 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6.
I20260808 20:14:36.368561 12205 leader_election.cc:302] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:36.359813 11896 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:57500 (request call id 312) took 9147 ms. Trace:
I20260808 20:14:36.361600 12249 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
I20260808 20:14:36.362655 12150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 293) took 4626 ms. Trace:
I20260808 20:14:36.379110 12150 rpcz_store.cc:276] 0808 20:14:31.736033 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.278039 (+4542006us) service_pool.cc:227] Handling call
0808 20:14:36.303292 (+ 25253us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.362310 (+ 59018us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:36.362347 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:36.362391 (+ 44us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:36.362424 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.362484 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:36.362600 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.343096 12244 raft_consensus.cc:1261] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:14:36.349866 13073 raft_consensus.cc:2454] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:36.367039 13016 leader_election.cc:302] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:36.294081 15116 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:36.287222 15477 raft_consensus.cc:2781] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:36.372923 12147 raft_consensus.cc:1226] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 7 LEADER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7. Ops: []
I20260808 20:14:36.357781 12144 raft_consensus.cc:1261] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 20:14:36.371883 12142 rpcz_store.cc:276] 0808 20:14:31.822513 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.342243 (+4519730us) service_pool.cc:227] Handling call
0808 20:14:36.342689 (+ 446us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.342975 (+ 286us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:36.343000 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:36.343035 (+ 35us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:36.343061 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.343107 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:36.343205 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.299006 14987 leader_election.cc:288] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:36.377586 12254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 379) took 1116 ms. Trace:
I20260808 20:14:36.390146 12147 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1323) took 4823 ms. Trace:
I20260808 20:14:36.382037 15192 consensus_queue.cc:260] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:36.382475 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:36.393471 12048 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:36.367697 13072 pending_rounds.cc:97] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260808 20:14:36.376963 12036 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 20:14:36.390013 12254 rpcz_store.cc:276] 0808 20:14:35.260757 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.242844 (+982087us) service_pool.cc:227] Handling call
0808 20:14:36.287075 (+ 44231us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.377339 (+ 90264us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.377533 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.360798 15375 raft_consensus.cc:2726] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:36.390699 12147 rpcz_store.cc:276] 0808 20:14:31.566527 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.273185 (+4706658us) service_pool.cc:227] Handling call
0808 20:14:36.296359 (+ 23174us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.388101 (+ 91742us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.390091 (+ 1990us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:36.395955 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 50039.8us
I20260808 20:14:36.376716 15426 raft_consensus.cc:2726] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:36.397221 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:36.279810 12137 rpcz_store.cc:276] 0808 20:14:35.236735 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:35.790157 (+553422us) service_pool.cc:227] Handling call
0808 20:14:36.262334 (+472177us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b540022fbd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:36.262468 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":377954688}
I20260808 20:14:36.370793 13077 rpcz_store.cc:276] 0808 20:14:34.312169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:35.840804 (+1528635us) service_pool.cc:227] Handling call
0808 20:14:35.841165 (+ 361us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:35.841523 (+ 358us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:35.841552 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:35.869950 (+ 28398us) log.cc:842] Serialized 26 byte log entry
0808 20:14:36.101369 (+231419us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:14:36.283988 (+182619us) log.cc:842] Serialized 14 byte log entry
0808 20:14:36.349263 (+ 65275us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.349348 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:36.349395 (+ 47us) raft_consensus.cc:1671] finished
0808 20:14:36.349409 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:36.349529 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25101,"prepare.queue_time_us":242438,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":56625,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":6563,"thread_start_us":396369,"threads_started":2,"wal-append.queue_time_us":112864}
I20260808 20:14:36.380141 15278 raft_consensus.cc:3037] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:36.389417 15116 raft_consensus.cc:3037] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:36.412381 12247 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 374) took 1701 ms. Trace:
W20260808 20:14:36.407986 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 233703us
I20260808 20:14:36.340377 11894 catalog_manager.cc:5495] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:36.412870 12247 rpcz_store.cc:276] 0808 20:14:34.706636 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.118391 (+1411755us) service_pool.cc:227] Handling call
0808 20:14:36.118830 (+ 439us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.119251 (+ 421us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 20:14:36.137848 (+ 18597us) log.cc:842] Serialized 14 byte log entry
0808 20:14:36.407725 (+269877us) log.cc:842] Serialized 14 byte log entry
0808 20:14:36.407870 (+ 145us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:36.407909 (+ 39us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:36.407944 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.408016 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:36.408115 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8272,"prepare.queue_time_us":15590,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":59,"raft.queue_time_us":23519,"thread_start_us":253291,"threads_started":2}
I20260808 20:14:36.392365 12144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1326) took 4788 ms. Trace:
I20260808 20:14:36.398233 12036 raft_consensus.cc:3037] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:14:36.425398 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 353535us
I20260808 20:14:36.421839 12000 leader_election.cc:302] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:36.425374 13073 raft_consensus.cc:1226] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 3. Current term is 4. Ops: []
W20260808 20:14:36.431186 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 337250us
I20260808 20:14:36.404989 13072 raft_consensus.cc:2866] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:36.409834 15477 raft_consensus.cc:686] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
W20260808 20:14:36.446520 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 150582us
I20260808 20:14:36.447505 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:36.447980 12147 raft_consensus.cc:1226] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260808 20:14:36.448112 12038 raft_consensus.cc:3037] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:36.428277 12138 raft_consensus.cc:1226] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:36.447731 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1399) took 1441 ms. Trace:
I20260808 20:14:36.436865 12101 leader_election.cc:302] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:36.449326 12105 leader_election.cc:302] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:36.449841 12147 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1617) took 4343 ms. Trace:
I20260808 20:14:36.454421 12147 rpcz_store.cc:276] 0808 20:14:32.106593 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.404329 (+4297736us) service_pool.cc:227] Handling call
0808 20:14:36.442308 (+ 37979us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:36.448341 (+ 6033us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.449789 (+ 1448us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.428298 15375 raft_consensus.cc:482] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:36.455320 12046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 450) took 1531 ms. Trace:
I20260808 20:14:36.443274 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:36.458598 15311 raft_consensus.cc:2781] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:36.459064 15311 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:36.459463 15311 raft_consensus.cc:3037] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:36.460042 12244 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260808 20:14:36.460986 15477 consensus_queue.cc:237] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:36.462126 13076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53290 (request call id 534) took 1453 ms. Trace:
I20260808 20:14:36.462563 13076 rpcz_store.cc:276] 0808 20:14:35.008353 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:35.873002 (+864649us) service_pool.cc:227] Handling call
0808 20:14:35.924233 (+ 51231us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:35.956746 (+ 32513us) raft_consensus.cc:1520] Early marking committed up to index 4
0808 20:14:36.257333 (+300587us) log.cc:842] Serialized 14 byte log entry
0808 20:14:36.455647 (+198314us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:36.461530 (+ 5883us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:36.461608 (+ 78us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.461723 (+ 115us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:36.461921 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5458,"prepare.queue_time_us":339251,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":38024,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":108,"thread_start_us":126204,"threads_started":2}
I20260808 20:14:36.465142 12040 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:36.465516 12146 raft_consensus.cc:1226] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260808 20:14:36.466187 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1619) took 4359 ms. Trace:
I20260808 20:14:36.458730 12249 raft_consensus.cc:1261] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260808 20:14:36.457268 12245 raft_consensus.cc:1261] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 20:14:36.466493 12146 rpcz_store.cc:276] 0808 20:14:32.107074 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.410281 (+4303207us) service_pool.cc:227] Handling call
0808 20:14:36.429610 (+ 19329us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:36.465928 (+ 36318us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.466121 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.467839 12142 raft_consensus.cc:1226] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260808 20:14:36.473981 13077 raft_consensus.cc:1226] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260808 20:14:36.466506 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49226 (request call id 457) took 1477 ms. Trace:
I20260808 20:14:36.452092 12041 rpcz_store.cc:276] 0808 20:14:35.005900 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.203651 (+1197751us) service_pool.cc:227] Handling call
0808 20:14:36.204109 (+ 458us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.204467 (+ 358us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 20:14:36.245688 (+ 41221us) log.cc:842] Serialized 14 byte log entry
0808 20:14:36.447416 (+201728us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:36.447467 (+ 51us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:36.447505 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.447562 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:36.447669 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20054,"prepare.queue_time_us":188,"prepare.run_cpu_time_us":119,"prepare.run_wall_time_us":119,"raft.queue_time_us":35415,"thread_start_us":127271,"threads_started":1}
I20260808 20:14:36.443979 14987 raft_consensus.cc:2726] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:36.421092 12144 rpcz_store.cc:276] 0808 20:14:31.604194 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.287156 (+4682962us) service_pool.cc:227] Handling call
0808 20:14:36.357406 (+ 70250us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.392063 (+ 34657us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.392308 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.453656 15192 raft_consensus.cc:3037] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:36.479671 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1616) took 4373 ms. Trace:
I20260808 20:14:36.465350 13075 raft_consensus.cc:1261] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEARNER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260808 20:14:36.498975 15311 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:36.507166 15311 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:36.510725 13077 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEARNER]: Advancing to term 6
W20260808 20:14:36.480924 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 73864.8us
I20260808 20:14:36.477491 12040 rpcz_store.cc:276] 0808 20:14:34.989304 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:35.457345 (+468041us) service_pool.cc:227] Handling call
0808 20:14:36.466310 (+1008965us) spinlock_profiling.cc:243] Waited 711 ms on lock 0x7b5400485dd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:36.466427 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1635929344}
I20260808 20:14:36.513880 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:36.484697 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 62951.9us
I20260808 20:14:36.460276 11994 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:36.516247 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 56083.4us
I20260808 20:14:36.516706 15245 raft_consensus.cc:2781] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:36.517037 11994 leader_election.cc:302] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:36.517273 15343 raft_consensus.cc:2781] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:36.466812 12137 raft_consensus.cc:1226] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6. Ops: []
W20260808 20:14:36.517755 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 236024us
I20260808 20:14:36.518072 12137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1621) took 4409 ms. Trace:
I20260808 20:14:36.518437 12137 rpcz_store.cc:276] 0808 20:14:32.108453 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.420491 (+4312038us) service_pool.cc:227] Handling call
0808 20:14:36.443984 (+ 23493us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.517818 (+ 73834us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.518014 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.468609 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:36.519729 13076 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:36.520030 12137 raft_consensus.cc:1261] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 20:14:36.520247 14987 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:36.487754 12142 rpcz_store.cc:276] 0808 20:14:32.106335 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.404159 (+4297824us) service_pool.cc:227] Handling call
0808 20:14:36.465545 (+ 61386us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:36.477656 (+ 12111us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.479619 (+ 1963us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.521715 12142 raft_consensus.cc:1226] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:36.522254 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1627) took 2973 ms. Trace:
I20260808 20:14:36.522552 12142 rpcz_store.cc:276] 0808 20:14:33.548648 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.521186 (+2972538us) service_pool.cc:227] Handling call
0808 20:14:36.521508 (+ 322us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.522046 (+ 538us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.522210 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.523669 12142 raft_consensus.cc:1226] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:36.524147 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1351) took 2180 ms. Trace:
I20260808 20:14:36.455906 12046 rpcz_store.cc:276] 0808 20:14:34.918786 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.106941 (+1188155us) service_pool.cc:227] Handling call
0808 20:14:36.107326 (+ 385us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.107862 (+ 536us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 20:14:36.157121 (+ 49259us) log.cc:842] Serialized 14 byte log entry
0808 20:14:36.424774 (+267653us) log.cc:842] Serialized 14 byte log entry
0808 20:14:36.449536 (+ 24762us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:36.449582 (+ 46us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:36.449616 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.449677 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:36.449779 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49217,"prepare.queue_time_us":25092,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":75,"raft.queue_time_us":48842,"thread_start_us":233403,"threads_started":2,"wal-append.queue_time_us":258499}
I20260808 20:14:36.427984 15446 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:36.480350 12138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 295) took 4726 ms. Trace:
I20260808 20:14:36.526430 12138 rpcz_store.cc:276] 0808 20:14:31.753660 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.287378 (+4533718us) service_pool.cc:227] Handling call
0808 20:14:36.404353 (+116975us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.455011 (+ 50658us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.478156 (+ 23145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.527683 15446 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:36.528317 13080 raft_consensus.cc:2454] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:36.492568 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:36.529417 15426 consensus_queue.cc:1046] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 20:14:36.497996 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:36.530802 15142 raft_consensus.cc:3032] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:36.526036 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:36.515206 12141 raft_consensus.cc:1226] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 6. Ops: []
I20260808 20:14:36.531736 12046 raft_consensus.cc:2454] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:36.532279 12141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1622) took 4107 ms. Trace:
I20260808 20:14:36.524413 12142 rpcz_store.cc:276] 0808 20:14:34.344054 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.523236 (+2179182us) service_pool.cc:227] Handling call
0808 20:14:36.523534 (+ 298us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.523945 (+ 411us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.524086 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.515054 12000 leader_election.cc:302] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:36.529738 13072 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:14:36.532560 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:36.534801 13075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53290 (request call id 538) took 1460 ms. Trace:
I20260808 20:14:36.484465 15332 raft_consensus.cc:2781] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:36.536473 12046 raft_consensus.cc:1261] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 20:14:36.532636 12141 rpcz_store.cc:276] 0808 20:14:32.425189 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.428124 (+4002935us) service_pool.cc:227] Handling call
0808 20:14:36.514955 (+ 86831us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.532028 (+ 17073us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.532220 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.537564 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:36.538028 15332 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:36.538125 12141 raft_consensus.cc:3037] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:36.538463 15332 raft_consensus.cc:3037] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:36.531037 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:36.535203 13075 rpcz_store.cc:276] 0808 20:14:35.074511 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.112312 (+1037801us) service_pool.cc:227] Handling call
0808 20:14:36.464983 (+352671us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.505145 (+ 40162us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.534575 (+ 29430us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b54006e1250. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d9a09 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:36.534733 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":810078464}
I20260808 20:14:36.541497 12041 raft_consensus.cc:1261] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 20:14:36.541770 15343 raft_consensus.cc:482] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:36.542243 15343 raft_consensus.cc:3037] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:36.542692 12041 raft_consensus.cc:1226] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4. Ops: []
W20260808 20:14:36.544106 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 118205us
I20260808 20:14:36.545014 12105 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:36.520673 12137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1340) took 2972 ms. Trace:
I20260808 20:14:36.545487 12137 rpcz_store.cc:276] 0808 20:14:33.548381 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.519230 (+2970849us) service_pool.cc:227] Handling call
0808 20:14:36.519640 (+ 410us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.520450 (+ 810us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.520629 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.546625 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:36.547127 12105 leader_election.cc:302] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:36.547272 12137 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
I20260808 20:14:36.547542 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:36.548540 15505 raft_consensus.cc:2726] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:36.520730 14987 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:36.549197 15505 raft_consensus.cc:2781] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:36.549690 15505 raft_consensus.cc:686] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:36.550470 14987 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:36.550670 15505 consensus_queue.cc:237] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:36.515841 15375 raft_consensus.cc:504] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:36.553463 11994 leader_election.cc:302] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:36.554271 15352 raft_consensus.cc:2726] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:36.555035 15375 leader_election.cc:288] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:36.556192 15506 raft_consensus.cc:2781] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:36.556576 15506 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:36.531193 15142 raft_consensus.cc:729] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:36.557971 15142 consensus_queue.cc:260] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:36.556914 15506 raft_consensus.cc:3037] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:36.559120 15142 raft_consensus.cc:3037] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:36.499082 13073 raft_consensus.cc:1261] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 20:14:36.565706 12205 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:36.566515 15506 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:36.567409 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:36.568504 11994 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:36.556744 12144 raft_consensus.cc:1226] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260808 20:14:36.569793 12146 raft_consensus.cc:1261] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 20:14:36.571851 13071 raft_consensus.cc:2454] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:14:36.572750 15205 raft_consensus.cc:2726] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:36.582695 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:53014 (request call id 1402) took 1661 ms. Trace:
W20260808 20:14:36.583449 15538 log.cc:924] Time spent T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
I20260808 20:14:36.583272 15402 consensus_queue.cc:1035] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 20:14:36.583949 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:36.584664 15096 raft_consensus.cc:2726] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:36.585016 13074 raft_consensus.cc:1226] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:36.585523 12152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 288) took 5521 ms. Trace:
I20260808 20:14:36.586135 11994 leader_election.cc:302] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:36.586411 15245 raft_consensus.cc:686] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:36.578938 15408 raft_consensus.cc:2726] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 20:14:36.583132 12041 rpcz_store.cc:276] 0808 20:14:34.918961 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.543404 (+1624443us) service_pool.cc:227] Handling call
0808 20:14:36.580273 (+ 36869us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.587387 15245 consensus_queue.cc:237] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:36.588322 11994 leader_election.cc:302] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:36.585035 12144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1626) took 3036 ms. Trace:
I20260808 20:14:36.589639 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:36.589682 12144 rpcz_store.cc:276] 0808 20:14:33.548361 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.497038 (+2948677us) service_pool.cc:227] Handling call
0808 20:14:36.556203 (+ 59165us) spinlock_profiling.cc:243] Waited 55.1 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:36.556420 (+ 217us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:36.584725 (+ 28305us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.584971 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":126696192}
I20260808 20:14:36.590656 12137 raft_consensus.cc:2379] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 5: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:36.591154 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:36.591730 12144 raft_consensus.cc:3037] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:36.584695 12142 raft_consensus.cc:2379] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:36.519666 13016 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:36.577854 13077 raft_consensus.cc:1261] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 LEARNER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 20:14:36.585892 12152 rpcz_store.cc:276] 0808 20:14:31.063794 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.125744 (+5061950us) service_pool.cc:227] Handling call
0808 20:14:36.126328 (+ 584us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:36.127008 (+ 680us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:36.127038 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:36.127826 (+ 788us) log.cc:842] Serialized 26 byte log entry
0808 20:14:36.134295 (+ 6469us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:36.134338 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.136794 (+ 2456us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:36.585321 (+448527us) raft_consensus.cc:1671] finished
0808 20:14:36.585343 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:36.585444 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1084,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":235,"thread_start_us":5842,"threads_started":1,"wal-append.queue_time_us":265405}
I20260808 20:14:36.582201 12038 raft_consensus.cc:2454] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
W20260808 20:14:36.573273 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 59887.5us
I20260808 20:14:36.527751 12138 raft_consensus.cc:1226] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 1. Current term is 3. Ops: []
I20260808 20:14:36.577359 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:36.605813 12043 raft_consensus.cc:2379] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 5: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:36.607832 12147 raft_consensus.cc:1226] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 5. Ops: []
I20260808 20:14:36.608426 12147 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 304) took 3074 ms. Trace:
I20260808 20:14:36.608704 12147 rpcz_store.cc:276] 0808 20:14:33.534125 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.475794 (+2941669us) service_pool.cc:227] Handling call
0808 20:14:36.592614 (+116820us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.608211 (+ 15597us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.608376 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.607519 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 305) took 3060 ms. Trace:
I20260808 20:14:36.610219 12146 rpcz_store.cc:276] 0808 20:14:33.547497 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.478437 (+2930940us) service_pool.cc:227] Handling call
0808 20:14:36.516873 (+ 38436us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.579762 (+ 62889us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.607249 (+ 27487us) spinlock_profiling.cc:243] Waited 52.4 ms on lock 0x7b54003fff54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:36.607438 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":120452864}
I20260808 20:14:36.604126 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:36.614377 12138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1352) took 2256 ms. Trace:
I20260808 20:14:36.612082 12101 leader_election.cc:302] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:36.594483 15342 raft_consensus.cc:2726] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:36.603894 15426 raft_consensus.cc:2781] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:36.619437 15452 raft_consensus.cc:2781] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:36.515868 11894 catalog_manager.cc:5495] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 2 to 5, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 5 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:36.626605 15452 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:36.629079 15452 raft_consensus.cc:3037] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:36.633663 13073 raft_consensus.cc:1261] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 20:14:36.635401 15506 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:36.594225 15360 consensus_queue.cc:1035] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260808 20:14:36.614789 15352 raft_consensus.cc:2726] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 20:14:36.638638 11994 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:36.636511 15452 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:36.616945 15311 raft_consensus.cc:2726] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:36.641150 15452 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
W20260808 20:14:36.629174 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 109619us
I20260808 20:14:36.642282 12152 raft_consensus.cc:1226] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:36.642896 12152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1641) took 1723 ms. Trace:
I20260808 20:14:36.643186 12152 rpcz_store.cc:276] 0808 20:14:34.921301 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.603309 (+1682008us) service_pool.cc:227] Handling call
0808 20:14:36.627054 (+ 23745us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.642674 (+ 15620us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.642845 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.644403 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:36.647804 12048 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
W20260808 20:14:36.648104 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 74978.7us
I20260808 20:14:36.650055 12142 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:14:36.650878 12147 raft_consensus.cc:1226] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:36.647333 12036 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:36.652426 11994 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:36.654408 12146 raft_consensus.cc:1226] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:36.655035 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1355) took 1701 ms. Trace:
I20260808 20:14:36.655474 12146 rpcz_store.cc:276] 0808 20:14:34.953561 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.615531 (+1661970us) service_pool.cc:227] Handling call
0808 20:14:36.651603 (+ 36072us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.654791 (+ 3188us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.654975 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.655411 12147 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1642) took 1711 ms. Trace:
I20260808 20:14:36.656440 12147 rpcz_store.cc:276] 0808 20:14:34.943867 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.609395 (+1665528us) service_pool.cc:227] Handling call
0808 20:14:36.650707 (+ 41312us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.655193 (+ 4486us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.655360 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.641360 15311 raft_consensus.cc:2726] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:14:36.627820 15426 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:36.665827 12146 raft_consensus.cc:1261] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 20:14:36.664363 15352 raft_consensus.cc:2726] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:36.668303 11994 leader_election.cc:302] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:36.662566 15360 consensus_queue.cc:1046] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20260808 20:14:36.658167 15205 raft_consensus.cc:2781] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:36.669940 15205 raft_consensus.cc:686] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:36.671226 15205 consensus_queue.cc:237] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:36.385476 11896 rpcz_store.cc:276] 0808 20:14:27.211440 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:27.211604 (+ 164us) service_pool.cc:227] Handling call
0808 20:14:36.359017 (+9147413us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 20:14:29.512255 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 20:14:29.562354 (+ 50099us) write_op.cc:432] Acquiring schema lock in shared mode
0808 20:14:29.562390 (+ 36us) write_op.cc:435] Acquired schema lock
0808 20:14:29.562404 (+ 14us) tablet.cc:660] Decoding operations
0808 20:14:29.753278 (+190874us) write_op.cc:620] Acquiring the partition lock for write op
0808 20:14:29.781587 (+ 28309us) write_op.cc:641] Partition lock acquired for write op
0808 20:14:29.786847 (+ 5260us) tablet.cc:683] Acquiring locks for 11 operations
0808 20:14:30.238544 (+451697us) tablet.cc:699] Row locks acquired
0808 20:14:30.238572 (+ 28us) write_op.cc:260] PREPARE: finished
0808 20:14:30.278470 (+ 39898us) write_op.cc:270] Start()
0808 20:14:30.285391 (+ 6921us) write_op.cc:276] Timestamp: P: 1786220070274385 usec, L: 0
0808 20:14:30.285414 (+ 23us) op_driver.cc:331] REPLICATION: starting
0808 20:14:30.389019 (+103605us) log.cc:842] Serialized 4388 byte log entry
0808 20:14:32.890946 (+2501927us) spinlock_profiling.cc:243] Waited 8.76 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:33.998311 (+1107365us) op_driver.cc:444] REPLICATION: finished
0808 20:14:34.297742 (+299431us) write_op.cc:301] APPLY: starting
0808 20:14:34.297831 (+ 89us) tablet.cc:1365] starting BulkCheckPresence
0808 20:14:34.301318 (+ 3487us) tablet.cc:1368] finished BulkCheckPresence
0808 20:14:34.301336 (+ 18us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 20:14:34.800145 (+498809us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 20:14:34.908473 (+108328us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0808 20:14:34.919980 (+ 11507us) write_op.cc:312] APPLY: finished
0808 20:14:34.961725 (+ 41745us) log.cc:842] Serialized 83 byte log entry
0808 20:14:35.170649 (+208924us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:14:35.212924 (+ 42275us) write_op.cc:454] Released schema lock
0808 20:14:35.212986 (+ 62us) write_op.cc:341] FINISH: Updating metrics
0808 20:14:35.321630 (+108644us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:14:35.358362 (+ 36732us) write_op.cc:454] Released schema lock
0808 20:14:35.609119 (+250757us) spinlock_profiling.cc:243] Waited 6.73 ms on lock 0x7b1800349c60. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96e2518b1 00007fa96e25e661 00007fa9653b2764 00007fa9653b5b11 00007fa9653b5ac9 00007fa9653b5a59 00007fa9653b5a21 00007fa9653b4d1d 00007fa96e616204 00007fa96e616039 00007fa965394dc6
0808 20:14:35.633691 (+ 24572us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
Metrics: {"mutex_wait_us":165416,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":284840,"apply.run_cpu_time_us":23143,"apply.run_wall_time_us":1130955,"num_ops":11,"prepare.queue_time_us":87537,"prepare.run_cpu_time_us":195990,"prepare.run_wall_time_us":1267561,"raft.queue_time_us":534937,"raft.run_cpu_time_us":24624,"raft.run_wall_time_us":123188,"replication_time_us":3677566,"spinlock_wait_cycles":326392704,"thread_start_us":990982,"threads_started":5,"wal-append.queue_time_us":734714,"wal-append.run_cpu_time_us":160836,"wal-append.run_wall_time_us":4007929}]]}
I20260808 20:14:36.666707 12036 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:36.694315 12152 raft_consensus.cc:1226] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:36.695806 12152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1646) took 1733 ms. Trace:
I20260808 20:14:36.696147 12152 rpcz_store.cc:276] 0808 20:14:34.962827 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.643860 (+1681033us) service_pool.cc:227] Handling call
0808 20:14:36.694140 (+ 50280us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.695574 (+ 1434us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.695753 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.698453 12147 raft_consensus.cc:1226] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 5. Ops: []
I20260808 20:14:36.656740 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:36.616772 12138 rpcz_store.cc:276] 0808 20:14:34.357513 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.527181 (+2169668us) service_pool.cc:227] Handling call
0808 20:14:36.527515 (+ 334us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.611566 (+ 84051us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.614322 (+ 2756us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.665179 15426 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:36.704803 12142 raft_consensus.cc:1226] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:36.705906 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:36.707288 13016 leader_election.cc:302] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:36.707178 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:36.708391 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1356) took 1724 ms. Trace:
I20260808 20:14:36.706807 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:36.708714 12038 raft_consensus.cc:2396] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260808 20:14:36.709434 12142 rpcz_store.cc:276] 0808 20:14:34.985264 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.659220 (+1673956us) service_pool.cc:227] Handling call
0808 20:14:36.692941 (+ 33721us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.708181 (+ 15240us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.708340 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:36.709652 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:36.714227 12152 raft_consensus.cc:2379] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:36.709735 12251 raft_consensus.cc:2454] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
W20260808 20:14:36.716032 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 47856.2us
I20260808 20:14:36.696339 15352 raft_consensus.cc:3037] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:36.716140 13079 raft_consensus.cc:2454] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:36.718029 12142 raft_consensus.cc:1226] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:36.697264 12243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1745) took 1193 ms. Trace:
I20260808 20:14:36.719735 13071 raft_consensus.cc:2379] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:36.720000 15452 raft_consensus.cc:2781] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:36.719908 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:36.722388 12147 raft_consensus.cc:1261] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 20:14:36.725639 12146 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260808 20:14:36.727707 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:36.702041 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:36.728331 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:36.733095 12152 raft_consensus.cc:1261] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 20:14:36.733793 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:36.733961 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:36.735505 12206 leader_election.cc:334] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 2.158s (SENT)
I20260808 20:14:36.735942 12206 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:36.738274 12043 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:36.743105 12142 raft_consensus.cc:1261] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 20:14:36.726987 15452 raft_consensus.cc:482] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:36.749689 11994 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:36.731801 15116 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:36.754647 12205 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:36.757148 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 94005.9us
I20260808 20:14:36.761492 12038 raft_consensus.cc:2354] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3.
I20260808 20:14:36.703318 15343 raft_consensus.cc:504] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:36.734980 15286 raft_consensus.cc:482] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:36.766700 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:36.767601 12039 raft_consensus.cc:2379] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 5: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:36.768648 15107 raft_consensus.cc:2726] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:36.735033 15322 raft_consensus.cc:482] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:36.770166 13016 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:36.772348 15311 raft_consensus.cc:2726] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:36.769188 15398 consensus_queue.cc:1035] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260808 20:14:36.772984 12141 raft_consensus.cc:2454] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:36.772730 15096 consensus_queue.cc:1046] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 20:14:36.773705 15343 leader_election.cc:288] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:36.773422 15286 raft_consensus.cc:504] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:36.775002 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:36.775489 14987 raft_consensus.cc:2726] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 20:14:36.789040 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:36.804456 12245 raft_consensus.cc:2454] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:36.789754 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:36.792496 15562 log.cc:924] Time spent T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20260808 20:14:36.793251 12040 raft_consensus.cc:2396] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 20:14:36.765471 15452 raft_consensus.cc:3037] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:36.760728 15477 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:36.758090 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:36.800174 12150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1615) took 4694 ms. Trace:
I20260808 20:14:36.788780 12147 raft_consensus.cc:1261] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 20:14:36.761736 15360 consensus_queue.cc:1046] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20260808 20:14:36.770351 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:36.806805 12249 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Advancing to term 8
W20260808 20:14:36.802207 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 69665.5us
I20260808 20:14:36.801373 13072 raft_consensus.cc:1226] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 5. Ops: []
I20260808 20:14:36.811357 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:36.774333 15322 raft_consensus.cc:504] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:36.816473 12251 raft_consensus.cc:3037] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:36.816847 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:36.817523 12245 raft_consensus.cc:3037] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:36.810334 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:36.815346 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 58784.2us
I20260808 20:14:36.820055 12039 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:36.809290 12150 rpcz_store.cc:276] 0808 20:14:32.105994 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.387459 (+4281465us) service_pool.cc:227] Handling call
0808 20:14:36.410588 (+ 23129us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.424293 (+ 13705us) raft_consensus.cc:1520] Early marking committed up to index 4
0808 20:14:36.710363 (+286070us) log.cc:842] Serialized 14 byte log entry
0808 20:14:36.799867 (+ 89504us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:36.799911 (+ 44us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:36.799949 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.800003 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:36.800119 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2402,"prepare.queue_time_us":39364,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":6739,"thread_start_us":303113,"threads_started":2}
W20260808 20:14:36.820689 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 78811.1us
I20260808 20:14:36.787592 15100 consensus_queue.cc:1035] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.022s
W20260808 20:14:36.823045 15551 log.cc:924] Time spent T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.118s user 0.000s sys 0.000s
I20260808 20:14:36.825372 12036 raft_consensus.cc:3037] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
W20260808 20:14:36.830224 15554 log.cc:924] Time spent T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.167s user 0.000s sys 0.001s
I20260808 20:14:36.832098 15204 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:36.840306 13015 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:36.834558 15116 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:36.815759 12138 raft_consensus.cc:3037] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:36.794611 15332 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:36.836256 15125 consensus_queue.cc:1035] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260808 20:14:36.836947 12241 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 6
I20260808 20:14:36.838479 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:36.808833 11872 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet cb8b79bb1c59483e93af813653a73a31 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica b9d94f35ea15421aa1d721201c782d8a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:36.811262 14987 raft_consensus.cc:2726] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:36.851617 15116 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260808 20:14:36.853384 12250 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1733) took 1381 ms. Trace:
I20260808 20:14:36.857559 12038 raft_consensus.cc:2454] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 7.
I20260808 20:14:36.859241 12248 raft_consensus.cc:1261] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260808 20:14:36.854332 12247 raft_consensus.cc:2932] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:36.836040 15286 leader_election.cc:288] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:36.861943 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 91855.3us
I20260808 20:14:36.863440 13016 leader_election.cc:302] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:36.864589 15565 log.cc:924] Time spent T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260808 20:14:36.838215 12144 raft_consensus.cc:2454] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:36.815660 15477 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:36.727227 12243 rpcz_store.cc:276] 0808 20:14:35.504050 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.363102 (+859052us) service_pool.cc:227] Handling call
0808 20:14:36.425142 (+ 62040us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.430997 (+ 5855us) raft_consensus.cc:1520] Early marking committed up to index 4
0808 20:14:36.435989 (+ 4992us) log.cc:842] Serialized 14 byte log entry
0808 20:14:36.626160 (+190171us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:36.627906 (+ 1746us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:36.648970 (+ 21064us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.657934 (+ 8964us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:36.697200 (+ 39266us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1143,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":2078,"raft.queue_time_us":13016,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":2072,"thread_start_us":159869,"threads_started":1,"wal-append.queue_time_us":187706}
I20260808 20:14:36.866096 15476 consensus_queue.cc:1046] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:14:36.866324 15278 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:36.867259 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:36.873486 12140 raft_consensus.cc:1226] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 1. Current term is 3. Ops: []
W20260808 20:14:36.892411 11872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 117931us
W20260808 20:14:36.894254 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 139683us
I20260808 20:14:36.898243 12249 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 8.
I20260808 20:14:36.896365 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:36.899410 13074 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:36.900183 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:36.866089 15342 consensus_queue.cc:1035] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260808 20:14:36.903676 15343 consensus_queue.cc:1046] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:36.905056 15161 raft_consensus.cc:2726] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:36.905682 15161 raft_consensus.cc:2726] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 20:14:36.906232 15161 raft_consensus.cc:2726] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:36.908315 13074 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:36.910356 15204 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:36.911015 12244 raft_consensus.cc:1226] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:36.912063 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:36.915257 12254 raft_consensus.cc:1261] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260808 20:14:36.915817 15278 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:36.916404 15278 raft_consensus.cc:3032] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:36.917176 12045 raft_consensus.cc:2379] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 5: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:36.918710 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:36.920153 12248 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1752) took 1326 ms. Trace:
I20260808 20:14:36.920565 12248 rpcz_store.cc:276] 0808 20:14:35.593701 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.294467 (+700766us) service_pool.cc:227] Handling call
0808 20:14:36.855627 (+561160us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.860536 (+ 4909us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.919939 (+ 59403us) spinlock_profiling.cc:243] Waited 494 ms on lock 0x7b5400086e50. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d9a09 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:36.920090 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1136738176}
I20260808 20:14:36.860281 12250 rpcz_store.cc:276] 0808 20:14:35.472037 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.227353 (+755316us) service_pool.cc:227] Handling call
0808 20:14:36.227799 (+ 446us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:36.228344 (+ 545us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:36.228371 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:36.284534 (+ 56163us) log.cc:842] Serialized 26 byte log entry
0808 20:14:36.415266 (+130732us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:36.415304 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.415363 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:36.853191 (+437828us) raft_consensus.cc:1671] finished
0808 20:14:36.853210 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:36.853332 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8777,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"thread_start_us":63832,"threads_started":1,"wal-append.queue_time_us":268915}
I20260808 20:14:36.926846 15477 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:36.927742 12146 raft_consensus.cc:3032] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Stepping down as leader of term 3
W20260808 20:14:36.928015 15557 log.cc:924] Time spent T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260808 20:14:36.928076 12146 raft_consensus.cc:729] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
W20260808 20:14:36.929608 13079 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53290 (request call id 533) took 1925 ms (client timeout 1911 ms). Trace:
W20260808 20:14:36.930001 13079 rpcz_store.cc:269] 0808 20:14:35.003903 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:35.150217 (+146314us) service_pool.cc:227] Handling call
0808 20:14:36.929361 (+1779144us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54001d9a54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:36.929542 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3184941440}
I20260808 20:14:36.932258 12101 leader_election.cc:302] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:36.933650 12205 leader_election.cc:302] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:36.938254 12043 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6.
I20260808 20:14:36.938943 12251 raft_consensus.cc:2454] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
W20260808 20:14:36.939672 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 81279us
I20260808 20:14:36.933710 12101 leader_election.cc:302] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
W20260808 20:14:36.943272 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 77518.7us
I20260808 20:14:36.916705 15278 raft_consensus.cc:729] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:36.948204 12245 raft_consensus.cc:2454] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:36.950062 12206 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:36.951537 12039 raft_consensus.cc:2454] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:36.950114 15286 raft_consensus.cc:2741] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260808 20:14:36.939225 12146 consensus_queue.cc:260] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:36.955467 12036 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:36.863370 15426 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:36.967234 13015 leader_election.cc:398] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Vote denied by peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747) with higher term. Message: Invalid argument: T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3.
I20260808 20:14:36.968729 13015 leader_election.cc:401] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260808 20:14:36.908752 15332 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:36.950601 15477 raft_consensus.cc:2781] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:36.982455 12146 raft_consensus.cc:3037] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:14:36.991848 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 174684us
I20260808 20:14:36.991782 13015 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:36.980141 15477 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:36.985972 15332 raft_consensus.cc:2726] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747) with higher term. Message: Invalid argument: T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3.
I20260808 20:14:37.006215 13079 raft_consensus.cc:1261] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 20:14:37.007833 15426 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:37.008991 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:36.995132 15278 consensus_queue.cc:260] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:37.010114 15278 raft_consensus.cc:3037] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:36.997666 15125 raft_consensus.cc:2781] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:36.996513 15477 raft_consensus.cc:3037] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:37.018044 12146 raft_consensus.cc:1261] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260808 20:14:37.018719 12146 pending_rounds.cc:76] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9: Aborting all ops after (but not including) 5
I20260808 20:14:37.019050 12146 pending_rounds.cc:97] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9: Aborting uncommitted NO_OP operation due to leader change: 3.6
W20260808 20:14:37.018321 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 68372.8us
I20260808 20:14:37.019505 12146 raft_consensus.cc:2866] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:37.017866 15332 raft_consensus.cc:2781] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:37.030268 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:37.033196 12245 raft_consensus.cc:2354] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 5. Current term is 6.
I20260808 20:14:37.018678 15476 consensus_queue.cc:1035] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260808 20:14:37.038563 12140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1318) took 5542 ms. Trace:
I20260808 20:14:37.038975 12140 rpcz_store.cc:276] 0808 20:14:31.496386 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.189303 (+4692917us) service_pool.cc:227] Handling call
0808 20:14:36.189696 (+ 393us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:36.891810 (+702114us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:37.038281 (+146471us) spinlock_profiling.cc:243] Waited 684 ms on lock 0x7b540030be54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:37.038470 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1572158720}
I20260808 20:14:37.042116 12241 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:14:37.042711 14987 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:37.043170 14987 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:37.044152 15125 raft_consensus.cc:482] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:37.044591 15125 raft_consensus.cc:3037] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:37.044950 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:37.047634 15452 raft_consensus.cc:504] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:37.042317 15556 log.cc:924] Time spent T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.249s user 0.004s sys 0.003s
I20260808 20:14:37.050009 12047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1407) took 1609 ms. Trace:
I20260808 20:14:37.051079 12048 raft_consensus.cc:1261] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 20:14:37.053669 15358 raft_consensus.cc:3032] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:37.054123 15358 raft_consensus.cc:729] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:37.055122 15358 consensus_queue.cc:260] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:37.056525 15358 raft_consensus.cc:3037] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:37.057440 15332 raft_consensus.cc:686] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:37.050325 12047 rpcz_store.cc:276] 0808 20:14:35.440731 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.215396 (+774665us) service_pool.cc:227] Handling call
0808 20:14:36.215777 (+ 381us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:36.216876 (+ 1099us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:36.243776 (+ 26900us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:36.280224 (+ 36448us) log.cc:842] Serialized 26 byte log entry
0808 20:14:36.608678 (+328454us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:36.608714 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.608778 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:37.041665 (+432887us) spinlock_profiling.cc:243] Waited 1.17 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:37.049714 (+ 8049us) raft_consensus.cc:1671] finished
0808 20:14:37.049733 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:37.049924 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2700160,"thread_start_us":230733,"threads_started":1,"wal-append.queue_time_us":426561}
I20260808 20:14:37.059163 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
W20260808 20:14:37.063427 12001 leader_election.cc:334] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Timed out: RequestConsensusVote RPC to 127.10.24.4:36901 timed out after 1.911s (SENT)
I20260808 20:14:37.020363 12138 raft_consensus.cc:2454] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:37.065923 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:37.066516 12251 raft_consensus.cc:3037] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:37.034261 15142 raft_consensus.cc:2781] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:37.070748 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
W20260808 20:14:37.073871 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 58110.3us
W20260808 20:14:37.076324 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 140530us
W20260808 20:14:37.084717 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 111401us
I20260808 20:14:37.070402 15142 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:37.092047 13072 raft_consensus.cc:1261] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260808 20:14:37.092530 15580 log.cc:924] Time spent T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.082s user 0.000s sys 0.001s
W20260808 20:14:37.092660 15568 log.cc:924] Time spent T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.098s user 0.001s sys 0.000s
I20260808 20:14:37.092079 15582 consensus_queue.cc:1035] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 20:14:37.096355 13077 raft_consensus.cc:1261] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 20:14:37.107445 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:37.096359 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:37.098341 15408 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260808 20:14:37.098743 15322 leader_election.cc:288] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:37.099489 15100 raft_consensus.cc:482] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:37.100143 15096 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:37.091096 12251 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:37.058830 15332 consensus_queue.cc:237] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:37.106283 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:37.105685 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:37.117071 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:37.109364 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:37.116225 13078 raft_consensus.cc:1226] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260808 20:14:37.117344 12140 raft_consensus.cc:2379] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 6: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:37.108290 13079 raft_consensus.cc:2379] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:37.111792 15506 raft_consensus.cc:3032] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:37.118708 12048 raft_consensus.cc:3037] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:37.104694 15142 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:37.118741 12146 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:14:37.122898 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:37.123610 12137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1643) took 2165 ms. Trace:
I20260808 20:14:37.123548 13079 raft_consensus.cc:1226] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260808 20:14:37.123613 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:37.126688 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:37.126231 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:37.126502 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:37.121671 12001 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:37.123554 13073 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:37.115064 15352 raft_consensus.cc:2726] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:37.127537 12146 raft_consensus.cc:2454] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:37.123809 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:37.143379 12048 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:37.121119 15506 raft_consensus.cc:729] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:37.119271 15100 raft_consensus.cc:504] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:37.149108 15506 consensus_queue.cc:260] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:37.150341 15506 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:37.150888 15100 leader_election.cc:288] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:37.143611 12101 leader_election.cc:302] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:37.146085 15578 consensus_queue.cc:1046] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20260808 20:14:37.137419 15375 consensus_queue.cc:1035] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20260808 20:14:37.144569 15343 consensus_queue.cc:1035] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260808 20:14:37.130673 15096 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:37.124039 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:37.154138 13014 leader_election.cc:398] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Vote denied by peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) with higher term. Message: Invalid argument: T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 5. Current term is 6.
I20260808 20:14:37.156170 13014 leader_election.cc:401] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260808 20:14:37.156752 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:37.124989 15452 leader_election.cc:288] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:37.159194 12101 leader_election.cc:302] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:37.160692 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:37.161213 15592 consensus_queue.cc:1035] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 20:14:37.161751 12150 raft_consensus.cc:2379] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:37.161785 12205 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:37.162585 12001 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:37.164685 12205 leader_election.cc:302] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:37.120329 15477 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:37.120450 15245 consensus_queue.cc:1035] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260808 20:14:37.173215 13014 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:37.162580 12144 raft_consensus.cc:2454] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:37.156455 15476 raft_consensus.cc:2781] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:37.175366 12142 raft_consensus.cc:2454] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 8.
I20260808 20:14:37.176084 12102 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:37.179340 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 43706us
I20260808 20:14:37.181787 12101 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:37.160006 15100 raft_consensus.cc:2781] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:37.188580 12152 raft_consensus.cc:2379] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:37.184700 15204 raft_consensus.cc:2781] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:37.201567 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:37.203140 15596 raft_consensus.cc:2781] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:14:37.202561 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:37.205235 12249 raft_consensus.cc:2454] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:37.186751 15161 raft_consensus.cc:2781] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:37.197474 15311 raft_consensus.cc:2781] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:37.190634 15402 raft_consensus.cc:2726] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:37.272456 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:37.289889 15505 raft_consensus.cc:3032] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:37.290416 15505 raft_consensus.cc:729] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:37.291368 15452 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:37.187333 15408 consensus_queue.cc:1046] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.043s
I20260808 20:14:37.291877 15452 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:37.291383 15505 consensus_queue.cc:260] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:37.293249 15408 raft_consensus.cc:3032] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:37.293520 15505 raft_consensus.cc:3037] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:37.294050 15452 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:37.198124 15352 consensus_queue.cc:1035] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260808 20:14:37.292315 14987 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:37.271351 12001 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:37.295521 15584 consensus_queue.cc:1035] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.091s
I20260808 20:14:37.296001 15161 raft_consensus.cc:686] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:37.199345 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:37.200116 15100 raft_consensus.cc:686] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:37.296967 15590 consensus_queue.cc:1035] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 20:14:37.297055 15161 consensus_queue.cc:237] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:37.298102 15100 consensus_queue.cc:237] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:37.299970 15477 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:37.124111 12137 rpcz_store.cc:276] 0808 20:14:34.958020 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:36.622622 (+1664602us) service_pool.cc:227] Handling call
0808 20:14:36.715484 (+ 92862us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:36.794337 (+ 78853us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:36.798218 (+ 3881us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:36.892113 (+ 93895us) log.cc:842] Serialized 26 byte log entry
0808 20:14:36.990085 (+ 97972us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:14:36.992593 (+ 2508us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:36.996034 (+ 3441us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:37.123380 (+127346us) raft_consensus.cc:1671] finished
0808 20:14:37.123407 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:37.123550 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19445,"raft.queue_time_us":22121,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":8444,"thread_start_us":69681,"threads_started":1,"wal-append.queue_time_us":90316}
I20260808 20:14:37.301383 15192 raft_consensus.cc:2781] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:37.301816 15192 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:37.302181 15192 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:37.302389 15577 consensus_queue.cc:1035] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:37.304112 15577 consensus_queue.cc:1035] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:37.305403 12001 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:37.306334 15342 raft_consensus.cc:2781] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:37.306839 15342 raft_consensus.cc:686] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:37.307714 15342 consensus_queue.cc:237] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:37.199563 15476 raft_consensus.cc:686] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:37.206596 15596 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:37.334717 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:37.335482 12142 raft_consensus.cc:2379] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
W20260808 20:14:37.336567 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 131850us
I20260808 20:14:37.198488 15125 raft_consensus.cc:504] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:37.277897 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:37.371914 13016 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:37.372951 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:37.375903 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:37.284960 15360 raft_consensus.cc:2781] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:37.272014 15204 raft_consensus.cc:686] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:37.272061 15205 raft_consensus.cc:2781] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:37.377635 12142 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:37.383442 13014 leader_election.cc:302] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:37.335886 15596 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:37.384299 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:37.347785 15116 raft_consensus.cc:482] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:37.388347 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:37.345799 15408 raft_consensus.cc:729] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
W20260808 20:14:37.388778 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 207066us
I20260808 20:14:37.393873 12152 raft_consensus.cc:2454] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
W20260808 20:14:37.389976 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 215367us
I20260808 20:14:37.351953 15311 raft_consensus.cc:686] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:37.230461 15286 consensus_queue.cc:1035] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.143s
I20260808 20:14:37.327064 15352 consensus_queue.cc:1035] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.069s
I20260808 20:14:37.405004 12101 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:37.389945 15398 raft_consensus.cc:2741] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260808 20:14:37.411669 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:37.414844 15573 raft_consensus.cc:482] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:37.415177 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 72712.1us
I20260808 20:14:37.393693 15360 raft_consensus.cc:686] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:37.435798 15402 raft_consensus.cc:2781] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 20:14:37.436177 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:37.436678 12045 raft_consensus.cc:1261] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260808 20:14:37.406405 15343 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:37.436216 12105 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:37.441071 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 105950us
I20260808 20:14:37.409538 15205 raft_consensus.cc:686] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:37.454061 13079 raft_consensus.cc:1226] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260808 20:14:37.455395 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:37.457118 15096 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:37.457235 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:37.429486 15192 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:37.457911 12152 raft_consensus.cc:3037] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:37.420819 15476 consensus_queue.cc:237] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:37.459313 15192 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:37.459860 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:37.460588 12047 raft_consensus.cc:2379] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 3: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:37.460783 12040 raft_consensus.cc:1261] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 20:14:37.461146 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:37.462234 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:37.462813 12137 raft_consensus.cc:3037] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:37.464579 15311 consensus_queue.cc:237] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:37.465134 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:37.446534 15408 consensus_queue.cc:260] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:37.465648 12152 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:37.466056 15408 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:37.452917 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:37.468055 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:37.469681 12249 raft_consensus.cc:2379] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 3: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:37.459584 15205 consensus_queue.cc:237] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:37.488060 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 91870.2us
I20260808 20:14:37.471671 12102 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:37.398252 15116 raft_consensus.cc:504] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:37.491508 12244 raft_consensus.cc:2396] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260808 20:14:37.487152 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:14:37.447257 12039 raft_consensus.cc:2932] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:37.492520 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:37.465749 12144 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
I20260808 20:14:37.499019 15585 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:37.499182 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:37.500204 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 64061us
I20260808 20:14:37.500406 12036 raft_consensus.cc:2379] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:37.500943 12105 leader_election.cc:302] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:37.460480 15278 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:37.504482 12038 raft_consensus.cc:1261] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260808 20:14:37.506022 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 96778.7us
I20260808 20:14:37.445320 15573 raft_consensus.cc:504] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:37.511067 12250 raft_consensus.cc:1261] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 20:14:37.499529 15585 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:37.515705 12250 raft_consensus.cc:1261] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260808 20:14:37.516232 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 44635.4us
I20260808 20:14:37.513644 12048 raft_consensus.cc:1226] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260808 20:14:37.516484 13016 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:37.518850 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:37.520414 12244 raft_consensus.cc:3037] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:37.448803 15343 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:37.521756 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 135787us
I20260808 20:14:37.500043 15402 raft_consensus.cc:686] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 8 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:37.504081 15245 raft_consensus.cc:2726] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:37.530207 15619 consensus_queue.cc:1035] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 20:14:37.538844 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:37.493446 15286 consensus_queue.cc:1035] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260808 20:14:37.540546 12250 raft_consensus.cc:1261] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260808 20:14:37.505718 15192 consensus_queue.cc:1046] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.022s
I20260808 20:14:37.521611 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:37.526464 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:37.544421 15107 raft_consensus.cc:2781] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:37.543372 15125 leader_election.cc:288] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:37.603223 12047 raft_consensus.cc:1261] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260808 20:14:37.542200 15096 raft_consensus.cc:2726] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 20:14:37.498430 15204 consensus_queue.cc:237] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:37.516680 15278 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:37.633143 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 77142.5us
I20260808 20:14:37.634292 12047 pending_rounds.cc:76] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting all ops after (but not including) 2
I20260808 20:14:37.634570 12047 pending_rounds.cc:97] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting uncommitted NO_OP operation due to leader change: 5.3
W20260808 20:14:37.633351 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 59129.5us
I20260808 20:14:37.634832 12047 raft_consensus.cc:2866] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 20:14:37.635354 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 158647us
I20260808 20:14:37.637358 12244 raft_consensus.cc:2454] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:37.646445 12043 raft_consensus.cc:1261] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260808 20:14:37.646884 12043 pending_rounds.cc:76] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting all ops after (but not including) 2
I20260808 20:14:37.647099 12043 pending_rounds.cc:97] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260808 20:14:37.647315 12043 raft_consensus.cc:2866] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 20:14:37.573764 15595 log.cc:924] Time spent T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.154s user 0.001s sys 0.000s
I20260808 20:14:37.653201 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:37.654640 12244 raft_consensus.cc:3037] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:37.558076 15571 raft_consensus.cc:2781] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:37.657178 12101 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:37.636405 12138 raft_consensus.cc:3037] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:37.653658 15125 consensus_queue.cc:1035] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260808 20:14:37.661023 15624 consensus_queue.cc:1035] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.106s
I20260808 20:14:37.661412 15116 leader_election.cc:288] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:37.662417 15164 raft_consensus.cc:2726] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:37.662743 15352 consensus_queue.cc:1046] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 20:14:37.663362 13015 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:37.664438 15352 raft_consensus.cc:3032] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:37.665262 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:37.558789 15360 consensus_queue.cc:237] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:37.665881 12038 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:14:37.559166 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:37.672569 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:37.674381 12205 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:37.677915 12244 raft_consensus.cc:2454] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:37.656209 15506 raft_consensus.cc:3032] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 20:14:37.574679 15100 consensus_queue.cc:1035] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.083s
W20260808 20:14:37.687937 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 187080us
I20260808 20:14:37.686780 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:37.673413 15358 consensus_meta.cc:287] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.605s user 0.010s sys 0.000s
I20260808 20:14:37.658802 15571 raft_consensus.cc:482] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:37.673825 12141 raft_consensus.cc:2379] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:37.664778 15352 raft_consensus.cc:729] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:37.689968 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:37.690127 12045 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:37.692203 12101 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:37.695031 12041 raft_consensus.cc:1226] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:37.713398 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:37.695421 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 49361.6us
I20260808 20:14:37.690163 12105 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:37.697387 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:37.716404 15585 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:37.718039 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:37.718652 13079 raft_consensus.cc:2454] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:37.719794 12207 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:37.728019 12137 raft_consensus.cc:2454] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:37.723794 12000 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:37.561844 15603 consensus_queue.cc:1035] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.176s
I20260808 20:14:37.689034 15506 raft_consensus.cc:729] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:37.731384 13015 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:37.755582 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:37.721751 15617 log.cc:924] Time spent T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045: Append to log took a long time: real 0.147s user 0.000s sys 0.001s
I20260808 20:14:37.697026 15571 raft_consensus.cc:3037] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:37.761839 12101 leader_election.cc:302] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
W20260808 20:14:37.765175 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 75082.4us
W20260808 20:14:37.753522 15595 log.cc:924] Time spent T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260808 20:14:37.731976 15245 raft_consensus.cc:2726] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:37.720834 15573 leader_election.cc:288] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:37.753814 15433 raft_consensus.cc:2726] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:37.770361 12105 leader_election.cc:302] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:37.756762 15585 raft_consensus.cc:2781] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:37.772689 12101 leader_election.cc:302] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:37.772837 12038 raft_consensus.cc:1261] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260808 20:14:37.774703 15636 consensus_queue.cc:1035] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260808 20:14:37.775063 15278 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:37.774833 15125 consensus_queue.cc:1035] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 20:14:37.775667 15615 raft_consensus.cc:2781] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:37.775967 12206 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:37.778013 15596 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:37.778797 15638 raft_consensus.cc:2726] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260808 20:14:37.780151 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 56438.6us
I20260808 20:14:37.837024 12000 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:37.770591 15582 consensus_queue.cc:1046] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20260808 20:14:37.785382 15477 raft_consensus.cc:2726] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:37.786741 13016 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:37.787967 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:37.687827 15402 consensus_queue.cc:237] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:37.512723 13074 raft_consensus.cc:2932] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:37.791879 12045 raft_consensus.cc:2454] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:14:37.767488 15591 raft_consensus.cc:2781] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:37.775010 15602 raft_consensus.cc:2781] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:37.761969 15096 consensus_queue.cc:1035] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
W20260808 20:14:37.841346 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 65460.3us
I20260808 20:14:37.818794 15433 raft_consensus.cc:2781] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:37.850292 12247 raft_consensus.cc:1226] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 7. Current term is 8. Ops: []
I20260808 20:14:37.842818 12205 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:37.767724 15164 consensus_queue.cc:1035] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.068s
I20260808 20:14:37.775698 15635 raft_consensus.cc:2781] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:37.837795 15343 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:37.808475 15286 consensus_queue.cc:1046] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20260808 20:14:37.858896 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:37.839184 15360 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
W20260808 20:14:37.861054 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 90766us
I20260808 20:14:37.862433 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:37.867314 12245 raft_consensus.cc:2454] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
I20260808 20:14:37.789985 15585 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:37.849660 15342 raft_consensus.cc:2726] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:37.835705 15116 consensus_queue.cc:1035] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.019s
I20260808 20:14:37.836118 15332 consensus_queue.cc:1035] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260808 20:14:37.843479 15591 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:14:37.899314 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 108405us
I20260808 20:14:37.865011 15433 raft_consensus.cc:686] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:37.865145 15584 raft_consensus.cc:2781] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:37.809333 15506 consensus_queue.cc:260] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:37.901875 12206 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:37.840435 15107 raft_consensus.cc:686] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:37.859802 15602 raft_consensus.cc:686] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:37.870632 15585 raft_consensus.cc:3037] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:37.934033 15591 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:37.838426 15352 consensus_queue.cc:260] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:37.946887 11872 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet cb8b79bb1c59483e93af813653a73a31 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica b9d94f35ea15421aa1d721201c782d8a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:37.870471 15100 consensus_queue.cc:1035] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.069s
W20260808 20:14:37.950176 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 163506us
I20260808 20:14:37.951479 13016 leader_election.cc:302] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:37.986032 15646 consensus_queue.cc:1035] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260808 20:14:37.986977 15596 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:37.944018 15358 raft_consensus.cc:2781] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:37.987690 15358 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:37.988037 15358 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
W20260808 20:14:37.990881 13073 consensus_meta.cc:287] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.852s user 0.005s sys 0.005s
I20260808 20:14:37.991330 13073 raft_consensus.cc:2454] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:14:37.992622 15322 raft_consensus.cc:2726] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) with higher term. Message: Invalid argument: T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 5. Current term is 6.
W20260808 20:14:37.993633 15639 log.cc:924] Time spent T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.174s user 0.001s sys 0.000s
I20260808 20:14:37.995186 15573 raft_consensus.cc:2726] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 20:14:37.995775 12138 raft_consensus.cc:2454] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:37.996798 15352 raft_consensus.cc:3037] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:37.998214 15433 consensus_queue.cc:237] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:37.956040 15506 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:37.948939 15584 raft_consensus.cc:686] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
W20260808 20:14:38.007670 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 58261.3us
I20260808 20:14:38.008459 12000 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:38.026959 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 75549.5us
I20260808 20:14:38.030952 12150 raft_consensus.cc:1261] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260808 20:14:37.898795 15638 consensus_queue.cc:1046] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.064s
W20260808 20:14:38.063973 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 73042.6us
I20260808 20:14:37.963196 15582 raft_consensus.cc:3032] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Stepping down as leader of term 2
W20260808 20:14:38.071014 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 62647.5us
W20260808 20:14:38.073248 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 209111us
I20260808 20:14:38.071887 15582 raft_consensus.cc:729] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:38.075264 15332 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:38.074245 15343 consensus_queue.cc:1046] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.027s
I20260808 20:14:38.078605 15615 raft_consensus.cc:686] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:38.135205 12206 leader_election.cc:302] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:38.075829 15646 raft_consensus.cc:2781] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:38.073300 15505 raft_consensus.cc:2726] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:38.073182 15161 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:38.147404 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 54899us
I20260808 20:14:38.107041 15446 raft_consensus.cc:2726] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 20:14:38.140445 15142 consensus_meta.cc:287] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 1.007s user 0.011s sys 0.000s
I20260808 20:14:38.153648 13014 leader_election.cc:302] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:38.134990 15286 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:38.149299 15360 raft_consensus.cc:2726] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 20:14:38.206743 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 99298.6us
I20260808 20:14:38.207484 12206 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:38.208926 12105 leader_election.cc:302] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:38.192939 15637 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:38.194545 15343 raft_consensus.cc:3032] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:38.070196 15584 consensus_queue.cc:237] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:38.192380 15433 raft_consensus.cc:482] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:38.194169 15596 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:38.211122 15644 log.cc:924] Time spent T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.273s user 0.001s sys 0.001s
I20260808 20:14:38.231019 12206 leader_election.cc:302] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:38.233920 15587 raft_consensus.cc:2726] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:38.198586 15161 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:38.234751 15670 log.cc:924] Time spent T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20260808 20:14:38.204346 15311 consensus_queue.cc:1035] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.031s
I20260808 20:14:38.236022 15161 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:38.221125 15343 raft_consensus.cc:729] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:38.237373 12206 leader_election.cc:302] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:38.155520 15332 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:38.154889 15653 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.300s
I20260808 20:14:38.281596 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:38.192948 15142 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:38.200685 15286 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:38.097342 15107 consensus_queue.cc:237] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:38.107592 15602 consensus_queue.cc:237] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:38.213785 15603 raft_consensus.cc:2726] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:38.208380 15205 consensus_queue.cc:1046] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.042s
I20260808 20:14:38.228399 15573 raft_consensus.cc:482] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:38.226686 15637 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:38.265678 15615 consensus_queue.cc:237] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:38.286764 15634 log.cc:924] Time spent T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.342s user 0.000s sys 0.001s
I20260808 20:14:38.295336 12043 raft_consensus.cc:1261] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260808 20:14:38.295677 15621 log.cc:924] Time spent T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.454s user 0.001s sys 0.000s
I20260808 20:14:38.299914 12039 raft_consensus.cc:1261] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 20:14:38.265331 15360 raft_consensus.cc:2781] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:38.313863 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:38.315558 15332 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:38.355505 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:38.298806 15204 consensus_queue.cc:1035] T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20260808 20:14:38.238781 15596 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:38.353174 15360 raft_consensus.cc:482] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:38.359030 15360 raft_consensus.cc:3037] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:38.359679 15596 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:38.281376 15638 consensus_queue.cc:1035] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.072s
I20260808 20:14:38.318012 15358 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:38.365314 15690 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:38.365792 15690 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:38.367097 15358 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:38.282084 15433 raft_consensus.cc:504] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:38.367699 15690 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:38.351043 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:38.367990 15591 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:38.237761 15343 consensus_queue.cc:260] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:38.369647 15343 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:38.349221 15573 raft_consensus.cc:504] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:38.318853 15311 raft_consensus.cc:482] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:38.371614 15573 leader_election.cc:288] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:38.371466 15311 raft_consensus.cc:504] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:38.225477 15582 consensus_queue.cc:260] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:38.373528 15622 log.cc:924] Time spent T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.593s user 0.001s sys 0.000s
W20260808 20:14:38.334766 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 51429us
I20260808 20:14:38.350328 15603 raft_consensus.cc:2726] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:38.373920 15360 raft_consensus.cc:504] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:38.375825 15360 leader_election.cc:288] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:38.376173 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:38.376806 13071 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 LEARNER]: Advancing to term 7
I20260808 20:14:38.376641 15687 consensus_queue.cc:1035] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 20:14:38.377400 12000 leader_election.cc:302] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:38.378129 15690 consensus_queue.cc:1035] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 20:14:38.378690 13073 raft_consensus.cc:1261] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 20:14:38.379397 15690 raft_consensus.cc:2781] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election won for term 5
W20260808 20:14:38.380874 15677 log.cc:924] Time spent T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.189s user 0.000s sys 0.001s
I20260808 20:14:38.382269 15690 raft_consensus.cc:686] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
W20260808 20:14:38.382489 15648 log.cc:924] Time spent T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.442s user 0.001s sys 0.000s
I20260808 20:14:38.345495 15664 raft_consensus.cc:2781] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:38.383438 15664 raft_consensus.cc:482] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:38.383136 15690 consensus_queue.cc:237] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:38.383805 15664 raft_consensus.cc:3037] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:38.386982 15582 raft_consensus.cc:3037] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
W20260808 20:14:38.335531 15571 consensus_meta.cc:287] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.548s user 0.010s sys 0.001s
I20260808 20:14:38.388691 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:38.389289 13079 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 7.
I20260808 20:14:38.389148 15571 raft_consensus.cc:504] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:38.387032 15433 leader_election.cc:288] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:38.391098 15571 leader_election.cc:288] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:38.357800 15679 log.cc:924] Time spent T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260808 20:14:38.393778 12207 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:38.394534 15603 raft_consensus.cc:2781] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:14:38.394901 15603 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:38.395251 15603 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:38.358139 15286 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:38.371042 15643 log.cc:924] Time spent T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.588s user 0.002s sys 0.000s
I20260808 20:14:38.378708 12000 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:38.401006 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:38.422786 12041 raft_consensus.cc:2396] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 20:14:38.424746 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:38.426337 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:38.426514 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 47870us
I20260808 20:14:38.427469 12000 leader_election.cc:302] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:38.430045 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:38.430627 12045 raft_consensus.cc:2454] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:38.333494 15658 consensus_queue.cc:1046] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.155s
I20260808 20:14:38.404697 15635 raft_consensus.cc:686] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:38.433084 15278 raft_consensus.cc:2781] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:38.433576 15278 raft_consensus.cc:686] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
W20260808 20:14:38.434219 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 80003.7us
I20260808 20:14:38.435683 12105 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:38.436545 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:38.436883 13015 leader_election.cc:302] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:38.373495 15311 leader_election.cc:288] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:38.438782 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:38.433717 15635 consensus_queue.cc:237] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:38.423493 15352 raft_consensus.cc:2781] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:14:38.436050 12243 raft_consensus.cc:1226] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 6. Current term is 7. Ops: []
I20260808 20:14:38.446955 15278 consensus_queue.cc:237] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:38.442955 15666 raft_consensus.cc:2781] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:38.449453 13071 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 LEARNER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 7.
I20260808 20:14:38.450927 15674 raft_consensus.cc:3032] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:38.444161 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:38.450495 15591 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:14:38.460966 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 164393us
I20260808 20:14:38.462966 15646 raft_consensus.cc:686] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:38.405215 15585 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:38.468600 15286 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:38.483021 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 127846us
W20260808 20:14:38.469273 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 315701us
I20260808 20:14:38.476424 13074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53296 (request call id 681) took 1065 ms. Trace:
I20260808 20:14:38.485544 13014 leader_election.cc:302] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:38.478395 15205 consensus_queue.cc:1225] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer b9d94f35ea15421aa1d721201c782d8a log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260808 20:14:38.479429 12039 raft_consensus.cc:1261] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260808 20:14:38.480602 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:38.453778 15352 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:38.495590 12041 raft_consensus.cc:1261] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 20:14:38.460624 15666 raft_consensus.cc:482] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:38.506834 12245 raft_consensus.cc:1261] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 20:14:38.488654 13079 raft_consensus.cc:1226] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:38.496775 12048 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:38.463732 15674 raft_consensus.cc:729] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:38.513114 12245 raft_consensus.cc:1261] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 20:14:38.442492 15332 consensus_queue.cc:1035] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260808 20:14:38.514942 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:38.478977 15096 raft_consensus.cc:2726] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:38.453722 15637 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:38.509629 15719 consensus_queue.cc:1035] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:38.515739 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:38.487876 15506 raft_consensus.cc:482] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:38.519323 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 282020us
I20260808 20:14:38.507417 15352 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:38.511688 15666 raft_consensus.cc:3037] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:38.503957 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:38.528507 12245 raft_consensus.cc:1261] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260808 20:14:38.529888 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:38.531728 15710 raft_consensus.cc:2726] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:38.511955 15205 consensus_queue.cc:1035] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260808 20:14:38.485277 13074 rpcz_store.cc:276] 0808 20:14:37.410970 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:37.416328 (+ 5358us) service_pool.cc:227] Handling call
0808 20:14:37.423987 (+ 7659us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:37.506646 (+ 82659us) raft_consensus.cc:1520] Early marking committed up to index 3
0808 20:14:38.075141 (+568495us) log.cc:842] Serialized 14 byte log entry
0808 20:14:38.433462 (+358321us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:38.433907 (+ 445us) log.cc:842] Serialized 26 byte log entry
0808 20:14:38.434033 (+ 126us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:38.475872 (+ 41839us) log.cc:842] Serialized 14 byte log entry
0808 20:14:38.476096 (+ 224us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:38.476167 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:38.476207 (+ 40us) raft_consensus.cc:1671] finished
0808 20:14:38.476222 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:38.476339 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":111965,"prepare.queue_time_us":16318,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":162457,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":12874,"spinlock_wait_cycles":8331648,"thread_start_us":166353,"threads_started":2,"wal-append.queue_time_us":339819}
I20260808 20:14:38.540984 13074 raft_consensus.cc:1261] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 20:14:38.513893 15286 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:38.522075 15664 raft_consensus.cc:504] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:38.550498 15699 consensus_queue.cc:1035] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.071s
I20260808 20:14:38.551653 15664 leader_election.cc:288] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:38.552456 15657 raft_consensus.cc:2726] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 20:14:38.553277 13073 raft_consensus.cc:1226] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:38.565425 12048 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 7.
I20260808 20:14:38.567188 12205 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:38.524689 15615 consensus_queue.cc:1035] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260808 20:14:38.524029 15311 consensus_queue.cc:1046] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260808 20:14:38.568717 12047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1898) took 1091 ms. Trace:
I20260808 20:14:38.570281 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:38.572348 12247 raft_consensus.cc:3037] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:38.541162 15666 raft_consensus.cc:504] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:38.570588 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:38.524958 15506 raft_consensus.cc:504] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:38.571521 12047 rpcz_store.cc:276] 0808 20:14:37.471176 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:37.471643 (+ 467us) service_pool.cc:227] Handling call
0808 20:14:37.504001 (+ 32358us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:37.644365 (+140364us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:38.562768 (+918403us) spinlock_profiling.cc:243] Waited 60.8 ms on lock 0x7b54000131d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:38.562928 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":139735936}
I20260808 20:14:38.577015 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:38.615314 12048 raft_consensus.cc:3037] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:38.615711 15615 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:38.601228 15358 raft_consensus.cc:2781] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election won for term 7
I20260808 20:14:38.616230 15615 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:38.618387 15615 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:38.618724 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:38.619349 12043 raft_consensus.cc:2454] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:38.619472 15710 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:38.619992 15710 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:38.621191 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:38.621781 12249 raft_consensus.cc:3037] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:38.621508 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:38.623828 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:38.624778 12250 raft_consensus.cc:2396] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260808 20:14:38.594128 12247 raft_consensus.cc:2454] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:38.627354 12048 raft_consensus.cc:2454] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:38.601843 13072 raft_consensus.cc:2396] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 20:14:38.622643 12251 raft_consensus.cc:2454] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:38.633875 12205 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:38.640303 13015 leader_election.cc:302] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:38.645519 12249 raft_consensus.cc:2454] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:38.641110 15358 raft_consensus.cc:686] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 7 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:38.647403 13014 leader_election.cc:302] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:38.648720 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:38.649180 13014 leader_election.cc:302] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:38.601274 15286 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:38.650345 13015 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:38.548851 15343 consensus_queue.cc:1035] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.095s
W20260808 20:14:38.708112 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 58998.5us
W20260808 20:14:38.711946 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 61686us
I20260808 20:14:38.710623 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:38.648862 15615 raft_consensus.cc:2781] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election won for term 5
W20260808 20:14:38.717927 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 190947us
I20260808 20:14:38.718505 12249 raft_consensus.cc:3037] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:38.707608 15591 raft_consensus.cc:2781] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:38.709412 15666 leader_election.cc:288] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:38.713923 15653 raft_consensus.cc:2726] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:38.729027 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:38.729691 12152 raft_consensus.cc:2454] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:38.724366 12206 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:38.720263 15358 consensus_queue.cc:237] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:38.725643 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:38.725962 15710 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:38.726176 15650 raft_consensus.cc:2781] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:38.723835 15245 raft_consensus.cc:482] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:38.789952 15245 raft_consensus.cc:504] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:38.726707 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
W20260808 20:14:38.719270 15648 log.cc:924] Time spent T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260808 20:14:38.739282 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:38.718384 15585 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:38.810243 15650 raft_consensus.cc:686] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:38.811168 15650 consensus_queue.cc:237] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:38.770198 12101 leader_election.cc:302] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:38.779891 13074 raft_consensus.cc:1261] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 20:14:38.549034 15674 consensus_queue.cc:260] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:38.581127 15646 consensus_queue.cc:237] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:38.721807 15591 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:38.740801 15719 consensus_queue.cc:1035] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260808 20:14:38.740646 15426 raft_consensus.cc:2726] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:38.725126 15692 consensus_queue.cc:1035] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.148s
I20260808 20:14:38.787492 12249 raft_consensus.cc:2454] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
W20260808 20:14:38.788519 15730 log.cc:924] Time spent T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20260808 20:14:38.719648 15205 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:38.793612 12150 raft_consensus.cc:1226] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 8 LEADER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260808 20:14:38.802374 15603 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:38.722157 15615 raft_consensus.cc:686] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:38.804687 12142 raft_consensus.cc:1261] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 20:14:38.807772 12043 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:38.793931 15506 leader_election.cc:288] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:38.809744 15723 log.cc:924] Time spent T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.239s user 0.000s sys 0.001s
I20260808 20:14:38.838369 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:38.791920 15245 leader_election.cc:288] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:38.897624 12152 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
W20260808 20:14:38.887460 15643 log.cc:924] Time spent T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.148s user 0.000s sys 0.000s
I20260808 20:14:38.839164 15278 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:38.888939 15591 raft_consensus.cc:3037] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:38.842671 12206 leader_election.cc:302] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:38.909626 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:38.911224 13015 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:38.812700 15653 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:38.922415 12152 raft_consensus.cc:1261] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 20:14:38.898283 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:38.912006 12245 raft_consensus.cc:2379] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 8: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:38.803485 12138 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:38.740048 15657 consensus_queue.cc:1035] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.192s
I20260808 20:14:38.839658 15696 raft_consensus.cc:2781] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:38.933691 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:38.937831 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:38.938241 15710 raft_consensus.cc:2781] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:38.938655 15710 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:38.941105 12142 raft_consensus.cc:1226] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:38.812731 15476 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:38.938997 15710 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:38.917937 15674 raft_consensus.cc:3037] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:38.977293 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:38.813485 13072 raft_consensus.cc:3037] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
W20260808 20:14:38.970672 15582 consensus_meta.cc:287] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.583s user 0.006s sys 0.005s
I20260808 20:14:38.894338 15205 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:38.984941 15736 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:38.923007 15733 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:38.929456 15584 raft_consensus.cc:2781] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:38.943065 15657 raft_consensus.cc:2726] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:38.943123 15603 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:38.970984 15311 raft_consensus.cc:482] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:39.051440 12045 raft_consensus.cc:1261] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 20:14:38.973124 15343 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:38.982589 15687 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:39.053520 12152 raft_consensus.cc:2454] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:38.985766 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:38.940338 15653 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:39.074106 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 304006us
I20260808 20:14:39.120324 12038 raft_consensus.cc:3032] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Stepping down as leader of term 2
W20260808 20:14:39.079988 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 237396us
I20260808 20:14:38.930039 15278 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:38.980434 15476 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:39.086769 15633 log.cc:924] Time spent T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 1.316s user 0.002s sys 0.008s
I20260808 20:14:38.995153 15584 raft_consensus.cc:482] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:38.996161 15736 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:38.996006 15733 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:38.991233 15615 consensus_queue.cc:237] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:39.230863 15678 log.cc:924] Time spent T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.874s user 0.000s sys 0.003s
W20260808 20:14:39.217654 15629 log.cc:924] Time spent T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 1.563s user 0.000s sys 0.007s
W20260808 20:14:39.111294 15685 log.cc:924] Time spent T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.717s user 0.003s sys 0.007s
I20260808 20:14:39.257594 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:39.126498 15506 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:39.244820 12102 leader_election.cc:302] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:39.250803 12038 raft_consensus.cc:729] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
W20260808 20:14:39.126034 15725 log.cc:924] Time spent T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.142s user 0.000s sys 0.000s
I20260808 20:14:39.219118 15597 raft_consensus.cc:3032] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:39.125818 13072 raft_consensus.cc:2454] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:39.283743 12249 raft_consensus.cc:2454] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
W20260808 20:14:39.228931 15681 log.cc:924] Time spent T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.882s user 0.002s sys 0.002s
I20260808 20:14:39.279462 15584 raft_consensus.cc:3037] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:39.289166 12206 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:39.292099 12045 raft_consensus.cc:1226] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260808 20:14:39.116994 15205 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:39.221230 15619 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:39.119072 15311 raft_consensus.cc:504] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:39.121935 15603 raft_consensus.cc:482] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:39.280207 15578 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:39.122324 15352 consensus_meta.cc:287] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.587s user 0.008s sys 0.000s
I20260808 20:14:39.288436 12038 consensus_queue.cc:260] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:39.318110 12142 raft_consensus.cc:1261] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 20:14:39.332619 12038 raft_consensus.cc:3037] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:39.334056 15311 leader_election.cc:288] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:39.121239 15343 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:39.088483 15358 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:39.292863 15597 raft_consensus.cc:729] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:39.341857 12038 raft_consensus.cc:1261] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260808 20:14:39.343823 12038 pending_rounds.cc:76] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting all ops after (but not including) 1
I20260808 20:14:39.264832 15657 consensus_queue.cc:1035] T 5e931b9df2c74729a3771d5f0734f559 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.060s
I20260808 20:14:39.292537 15506 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:39.328457 15205 raft_consensus.cc:482] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:39.417147 15205 raft_consensus.cc:504] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:39.420142 15205 leader_election.cc:288] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:39.338376 15603 raft_consensus.cc:504] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:39.359650 13072 raft_consensus.cc:1261] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 20:14:39.361560 15653 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:39.340713 15352 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:39.359835 15343 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:39.366734 15713 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:39.368991 15597 consensus_queue.cc:260] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:39.338289 12243 raft_consensus.cc:1226] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 7. Current term is 8. Ops: [5.3-7.4]
I20260808 20:14:39.337971 15358 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:39.381699 12248 raft_consensus.cc:1261] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 20:14:39.260824 15687 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:39.391232 15699 consensus_queue.cc:1035] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:14:39.401690 15476 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:39.408320 12244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1895) took 1462 ms. Trace:
I20260808 20:14:39.411613 12038 pending_rounds.cc:97] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260808 20:14:39.425310 15736 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:39.359055 12241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 391) took 2291 ms. Trace:
I20260808 20:14:39.436964 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:39.356534 15578 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:39.457906 12038 raft_consensus.cc:2866] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:39.457854 12254 raft_consensus.cc:1261] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 20:14:39.495641 12254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1890) took 1976 ms. Trace:
W20260808 20:14:39.442514 15696 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.020s sys 0.046s
W20260808 20:14:39.496181 15696 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.563s user 0.020s sys 0.046s
I20260808 20:14:39.495055 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1905) took 1559 ms. Trace:
I20260808 20:14:39.496626 12038 rpcz_store.cc:276] 0808 20:14:37.935388 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:37.935499 (+ 111us) service_pool.cc:227] Handling call
0808 20:14:37.935808 (+ 309us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:39.120229 (+1184421us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b540024a020. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa96826a730 00007fa9682dc300 00007fa9682dd371 00007fa9682d9edf 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc
0808 20:14:39.494789 (+374560us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:39.494998 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":346144640}
I20260808 20:14:39.495977 12254 rpcz_store.cc:276] 0808 20:14:37.519375 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:37.541594 (+ 22219us) service_pool.cc:227] Handling call
0808 20:14:39.457332 (+1915738us) spinlock_profiling.cc:243] Waited 1.08 ms on lock 0x7b58001a1398. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:39.457545 (+ 213us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:39.495396 (+ 37851us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:39.495588 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2493696}
I20260808 20:14:39.339102 15619 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:39.457050 12244 rpcz_store.cc:276] 0808 20:14:37.945506 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:37.945709 (+ 203us) service_pool.cc:227] Handling call
0808 20:14:37.959592 (+ 13883us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:37.981765 (+ 22173us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:37.981797 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:37.997714 (+ 15917us) log.cc:842] Serialized 26 byte log entry
0808 20:14:38.138930 (+141216us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:14:38.138968 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:38.139031 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:39.227208 (+1088177us) spinlock_profiling.cc:243] Waited 3.33 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:39.378798 (+151590us) raft_consensus.cc:1671] finished
0808 20:14:39.378819 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:39.408251 (+ 29432us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10178,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":166,"spinlock_wait_cycles":7668608,"thread_start_us":140860,"threads_started":1,"wal-append.queue_time_us":223840}
I20260808 20:14:39.433055 15506 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:39.486311 13015 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.025s sys 0.044s
I20260808 20:14:39.458789 15597 raft_consensus.cc:3037] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
W20260808 20:14:39.509320 13015 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.591s user 0.025s sys 0.044s
I20260808 20:14:39.459065 15352 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:39.503750 15192 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260808 20:14:39.487469 15278 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:39.512352 15619 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:39.509637 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 598468us
I20260808 20:14:39.539521 15653 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:39.540603 15653 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:39.542563 12137 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.643s user 0.021s sys 0.054s
W20260808 20:14:39.542915 12137 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s user 0.021s sys 0.054s
I20260808 20:14:39.512365 15578 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:39.500308 15696 raft_consensus.cc:686] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
W20260808 20:14:39.535617 15710 consensus_meta.cc:287] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.549s user 0.006s sys 0.005s
I20260808 20:14:39.568822 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:39.573925 15603 leader_election.cc:288] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:39.494215 12241 rpcz_store.cc:276] 0808 20:14:37.067234 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:37.068348 (+ 1114us) service_pool.cc:227] Handling call
0808 20:14:37.090692 (+ 22344us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:37.387973 (+297281us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:37.387998 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:37.445211 (+ 57213us) log.cc:842] Serialized 26 byte log entry
0808 20:14:37.538276 (+ 93065us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:37.538306 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:37.538365 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:39.127355 (+1588990us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:39.338368 (+211013us) raft_consensus.cc:1671] finished
0808 20:14:39.341015 (+ 2647us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:39.358966 (+ 17951us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":81137,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":94,"spinlock_wait_cycles":24396544,"thread_start_us":279040,"threads_started":2,"wal-append.queue_time_us":51559}
W20260808 20:14:39.579814 15591 consensus_meta.cc:287] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.662s user 0.004s sys 0.004s
I20260808 20:14:39.494917 15713 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:39.590894 12103 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.554s user 0.017s sys 0.044s
W20260808 20:14:39.603849 12103 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.758s user 0.017s sys 0.045s
I20260808 20:14:39.591975 15452 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:39.604617 15452 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:39.607308 12146 raft_consensus.cc:1226] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260808 20:14:39.592864 15710 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:39.599215 15358 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:39.559250 15733 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:39.609458 15591 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:39.616081 15587 consensus_queue.cc:1035] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 20:14:39.617043 15587 consensus_queue.cc:1225] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer 4cd46f13bceb4c3c9ef71e755e17b995 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260808 20:14:39.633622 12146 raft_consensus.cc:1226] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:39.609094 15603 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:39.622030 15700 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:39.630824 15591 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:39.661001 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:39.660691 15687 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:39.665879 13075 raft_consensus.cc:1226] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 LEARNER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260808 20:14:39.666913 13075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53290 (request call id 562) took 1167 ms. Trace:
I20260808 20:14:39.667480 13075 rpcz_store.cc:276] 0808 20:14:38.499274 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:38.499415 (+ 141us) service_pool.cc:227] Handling call
0808 20:14:39.665233 (+1165818us) spinlock_profiling.cc:243] Waited 6.37 ms on lock 0x7b5800005198. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:39.665552 (+ 319us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:39.666431 (+ 879us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:39.666829 (+ 398us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14653824}
W20260808 20:14:39.649381 12138 consensus_meta.cc:287] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.665s user 0.012s sys 0.000s
I20260808 20:14:39.669275 15710 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:39.773357 15452 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:39.649004 15743 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:39.663761 13077 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53290 (request call id 559) took 1165 ms. Trace:
I20260808 20:14:39.829633 13077 rpcz_store.cc:276] 0808 20:14:38.497907 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:38.499453 (+ 1546us) service_pool.cc:227] Handling call
0808 20:14:39.660806 (+1161353us) spinlock_profiling.cc:243] Waited 6.02 ms on lock 0x7b5800005198. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:39.663700 (+ 2894us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13835520}
W20260808 20:14:39.701725 15691 log.cc:924] Time spent T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 1.225s user 0.022s sys 0.007s
I20260808 20:14:39.756386 15713 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:39.832612 15743 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:39.751279 15736 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:39.657760 15603 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:39.831444 15433 raft_consensus.cc:3032] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 20:14:39.853083 15433 raft_consensus.cc:729] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:39.853302 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:53014 (request call id 1403) took 1468 ms. Trace:
I20260808 20:14:39.853675 12040 rpcz_store.cc:276] 0808 20:14:38.384886 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:38.393825 (+ 8939us) service_pool.cc:227] Handling call
0808 20:14:39.850871 (+1457046us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b5400464bd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825faf0 00007fa9682e2f2a 00007fa96d21fc6e 00007fa966916807 00007fa966916792 00007fa9669166c1 00007fa96691663c 00007fa9669158f2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:39.853162 (+ 2291us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":324393728}
I20260808 20:14:39.854110 15433 consensus_queue.cc:260] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:39.855510 15433 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:39.817838 15653 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:39.672257 15584 raft_consensus.cc:504] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:39.757174 15358 raft_consensus.cc:2781] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 20:14:39.822400 12138 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
I20260808 20:14:39.905938 15624 raft_consensus.cc:3032] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:39.906584 15624 raft_consensus.cc:729] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:39.913436 12245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53030 (request call id 408) took 1383 ms. Trace:
I20260808 20:14:39.913861 12245 rpcz_store.cc:276] 0808 20:14:38.529581 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:38.529683 (+ 102us) service_pool.cc:227] Handling call
0808 20:14:39.913141 (+1383458us) spinlock_profiling.cc:243] Waited 377 ms on lock 0x7b54000bbdd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:39.913277 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":867414400}
I20260808 20:14:39.877357 15358 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:39.903461 15571 raft_consensus.cc:482] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:39.939021 15571 raft_consensus.cc:504] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:39.939359 15603 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:39.687167 15696 consensus_queue.cc:237] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:39.942844 15571 leader_election.cc:288] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:39.928472 15683 log.cc:924] Time spent T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 1.530s user 0.001s sys 0.004s
I20260808 20:14:39.907938 15624 consensus_queue.cc:260] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:39.944156 15624 raft_consensus.cc:3037] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:39.949836 12046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1904) took 2016 ms. Trace:
I20260808 20:14:39.950214 12046 rpcz_store.cc:276] 0808 20:14:37.933942 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:37.935035 (+ 1093us) service_pool.cc:227] Handling call
0808 20:14:37.943684 (+ 8649us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 20:14:37.960661 (+ 16977us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:37.960691 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 20:14:37.963069 (+ 2378us) log.cc:842] Serialized 52 byte log entry
0808 20:14:38.152113 (+189044us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:14:38.152149 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:38.152218 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:39.892034 (+1739816us) spinlock_profiling.cc:243] Waited 62.5 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:39.945121 (+ 53087us) raft_consensus.cc:1671] finished
0808 20:14:39.945143 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:39.949761 (+ 4618us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1603,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":37895,"spinlock_wait_cycles":143798528,"thread_start_us":87507,"threads_started":1,"wal-append.queue_time_us":319708}
I20260808 20:14:39.917567 12138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1666) took 1185 ms. Trace:
I20260808 20:14:39.878042 15736 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:39.662916 15700 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:39.937412 15573 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:39.945866 15358 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Advancing to term 9
W20260808 20:14:39.985039 12140 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.011s user 0.008s sys 0.036s
W20260808 20:14:40.023252 12140 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.049s user 0.008s sys 0.036s
W20260808 20:14:39.985337 15602 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.049s user 0.000s sys 0.063s
W20260808 20:14:39.987838 15646 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.050s user 0.040s sys 0.029s
W20260808 20:14:40.042989 12039 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.727s user 0.010s sys 0.029s
W20260808 20:14:39.991096 13016 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.078s user 0.027s sys 0.007s
W20260808 20:14:40.045336 13016 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.132s user 0.027s sys 0.007s
W20260808 20:14:40.045434 12039 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.730s user 0.010s sys 0.029s
W20260808 20:14:39.991895 12105 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.051s user 0.018s sys 0.015s
W20260808 20:14:40.048638 12105 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.108s user 0.018s sys 0.015s
W20260808 20:14:40.011646 12144 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.771s user 0.067s sys 0.103s
W20260808 20:14:39.980166 15650 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.041s user 0.015s sys 0.033s
W20260808 20:14:40.073067 12000 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.765s user 0.021s sys 0.018s
W20260808 20:14:40.077437 12000 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.769s user 0.021s sys 0.018s
I20260808 20:14:40.077834 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:40.042737 15602 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.106s user 0.000s sys 0.063s
I20260808 20:14:40.088634 13077 raft_consensus.cc:2396] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260808 20:14:40.087913 13080 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.762s user 0.009s sys 0.029s
W20260808 20:14:40.043794 15646 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.106s user 0.041s sys 0.029s
W20260808 20:14:40.090541 12043 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.750s user 0.010s sys 0.019s
W20260808 20:14:40.071429 12144 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.833s user 0.068s sys 0.103s
W20260808 20:14:40.089946 13080 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.764s user 0.010s sys 0.029s
W20260808 20:14:40.102067 12043 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.762s user 0.010s sys 0.019s
W20260808 20:14:40.091087 12101 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.838s user 0.013s sys 0.029s
W20260808 20:14:40.126077 12101 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.873s user 0.013s sys 0.029s
W20260808 20:14:40.098107 13078 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.838s user 0.008s sys 0.019s
W20260808 20:14:40.127532 13078 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.872s user 0.009s sys 0.020s
W20260808 20:14:40.099961 15667 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.720s user 0.000s sys 0.006s
W20260808 20:14:40.071252 15650 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.132s user 0.015s sys 0.034s
I20260808 20:14:39.973636 12138 rpcz_store.cc:276] 0808 20:14:38.725012 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:38.725338 (+ 326us) service_pool.cc:227] Handling call
0808 20:14:39.909104 (+1183766us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:40.041033 15573 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:40.109095 15727 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:40.106848 12001 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.914s user 0.007s sys 0.021s
W20260808 20:14:40.200752 12001 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.008s user 0.007s sys 0.021s
W20260808 20:14:40.201637 12142 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.013s sys 0.006s
W20260808 20:14:40.203253 12142 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.633s user 0.013s sys 0.007s
I20260808 20:14:40.091300 12000 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:40.090879 15739 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:40.110358 15578 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:40.112454 15713 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:40.109493 15748 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
W20260808 20:14:40.159616 12102 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.872s user 0.010s sys 0.020s
W20260808 20:14:40.160667 15667 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.754s user 0.000s sys 0.007s
I20260808 20:14:40.130126 15602 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:40.111922 15741 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:40.169101 15690 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.782s user 0.005s sys 0.017s
W20260808 20:14:40.201975 12206 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.884s user 0.010s sys 0.009s
W20260808 20:14:40.202219 12041 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.858s user 0.005s sys 0.013s
W20260808 20:14:40.369287 12205 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.118s user 0.029s sys 0.018s
W20260808 20:14:40.503674 15754 log.cc:924] Time spent T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.207s user 0.001s sys 0.000s
W20260808 20:14:40.436975 12251 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.982s user 0.012s sys 0.011s
W20260808 20:14:40.326921 13016 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.600s (SENT) [suppressed 36 similar messages]
W20260808 20:14:40.416441 12247 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.134s user 0.029s sys 0.026s
W20260808 20:14:40.575055 12205 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.334s user 0.042s sys 0.034s
W20260808 20:14:40.574349 12206 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.256s user 0.012s sys 0.011s
W20260808 20:14:40.437104 11994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.310s user 0.002s sys 0.015s
W20260808 20:14:40.586827 11994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.460s user 0.003s sys 0.016s
W20260808 20:14:40.503461 12249 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.147s user 0.018s sys 0.031s
W20260808 20:14:40.363955 12137 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.780s user 0.015s sys 0.024s
W20260808 20:14:40.585633 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 1.34466e+06us
I20260808 20:14:40.263136 12046 raft_consensus.cc:1261] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260808 20:14:40.585222 12247 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.303s user 0.042s sys 0.036s
W20260808 20:14:40.437321 12207 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.913s user 0.008s sys 0.012s
W20260808 20:14:40.585649 12041 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.244s user 0.007s sys 0.015s
W20260808 20:14:40.436777 12150 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.001s sys 0.014s
W20260808 20:14:40.436496 12147 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.028s user 0.003s sys 0.016s
I20260808 20:14:40.277904 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:40.586890 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 1.2977e+06us
W20260808 20:14:40.686034 15358 consensus_meta.cc:287] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.634s user 0.009s sys 0.000s
W20260808 20:14:40.504192 12141 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.123s user 0.010s sys 0.038s
W20260808 20:14:40.684639 15624 consensus_meta.cc:287] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.740s user 0.012s sys 0.000s
I20260808 20:14:40.214141 15727 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:40.691777 14987 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:40.203668 15674 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:40.503815 15592 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.143s user 0.004s sys 0.043s
I20260808 20:14:40.692317 14987 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:40.474531 13071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53290 (request call id 558) took 1976 ms. Trace:
I20260808 20:14:40.198179 15733 raft_consensus.cc:482] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:40.693714 15733 raft_consensus.cc:504] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:40.689394 15358 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:40.695186 14987 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:40.695567 15733 leader_election.cc:288] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:40.696026 15733 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:40.587227 12249 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.231s user 0.019s sys 0.031s
I20260808 20:14:40.199021 15584 leader_election.cc:288] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:40.696372 15358 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 9 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:40.693027 12137 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.109s user 0.015s sys 0.067s
I20260808 20:14:40.694000 15789 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:40.307134 15697 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:40.698139 12206 leader_election.cc:302] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:40.220827 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:41.330773 12150 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.512s user 0.008s sys 0.021s
W20260808 20:14:40.794268 13080 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.009s sys 0.011s
W20260808 20:14:41.331426 13080 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.201s user 0.019s sys 0.021s
W20260808 20:14:40.533030 15775 log.cc:924] Time spent T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.162s user 0.001s sys 0.000s
I20260808 20:14:40.255496 15653 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:40.587260 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 496187us
I20260808 20:14:40.412297 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:40.736580 12147 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.330s user 0.006s sys 0.018s
W20260808 20:14:40.945705 12039 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.819s user 0.018s sys 0.010s
W20260808 20:14:41.339452 12039 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.230s user 0.026s sys 0.015s
W20260808 20:14:40.970568 13077 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.705s user 0.000s sys 0.019s
W20260808 20:14:40.573067 15690 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.186s user 0.006s sys 0.022s
I20260808 20:14:40.604907 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:40.549882 15747 log.cc:924] Time spent T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.196s user 0.001s sys 0.000s
I20260808 20:14:40.194337 15646 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:40.647904 15771 raft_consensus.cc:2781] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260808 20:14:41.329344 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 631239us
W20260808 20:14:40.588985 12102 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.301s user 0.014s sys 0.022s
W20260808 20:14:41.280817 12249 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.005s sys 0.006s
W20260808 20:14:41.340623 13016 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.764s user 0.003s sys 0.011s
W20260808 20:14:41.337105 12251 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.882s user 0.012s sys 0.029s
I20260808 20:14:41.349411 12046 raft_consensus.cc:2454] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
W20260808 20:14:40.547770 15675 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.901s user 0.007s sys 0.022s
W20260808 20:14:40.436631 15596 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.041s user 0.004s sys 0.021s
W20260808 20:14:40.436705 13073 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.800s user 0.007s sys 0.012s
W20260808 20:14:40.437254 13076 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.841s user 0.009s sys 0.011s
W20260808 20:14:40.277518 12203 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.815s user 0.016s sys 0.019s
W20260808 20:14:41.425772 12203 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.218s user 0.022s sys 0.041s
I20260808 20:14:40.325467 15736 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:40.236200 15571 raft_consensus.cc:482] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:40.255981 15746 raft_consensus.cc:482] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:40.353086 15142 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:14:41.340167 13077 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.074s user 0.000s sys 0.027s
W20260808 20:14:40.588994 12207 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.064s user 0.009s sys 0.012s
I20260808 20:14:41.334134 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49226 (request call id 492) took 1114 ms. Trace:
W20260808 20:14:40.774015 11872 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet cb8b79bb1c59483e93af813653a73a31 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica b9d94f35ea15421aa1d721201c782d8a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
W20260808 20:14:40.454071 15752 log.cc:924] Time spent T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.193s user 0.000s sys 0.001s
W20260808 20:14:40.437067 15673 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 2.088s user 0.000s sys 0.024s
W20260808 20:14:41.231124 15790 log.cc:924] Time spent T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.359s user 0.001s sys 0.001s
W20260808 20:14:40.733712 12141 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.352s user 0.015s sys 0.043s
W20260808 20:14:40.436857 12146 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.680s user 0.005s sys 0.013s
W20260808 20:14:41.437954 12146 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.681s user 0.009s sys 0.028s
W20260808 20:14:41.417394 15596 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 3.022s user 0.007s sys 0.032s
W20260808 20:14:41.309881 15755 log.cc:924] Time spent T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.388s user 0.000s sys 0.001s
I20260808 20:14:40.236771 15758 raft_consensus.cc:482] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
W20260808 20:14:41.166049 15638 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.784s user 0.014s sys 0.043s
I20260808 20:14:40.264770 13075 raft_consensus.cc:1226] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6. Ops: []
W20260808 20:14:40.696949 15592 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.336s user 0.008s sys 0.046s
I20260808 20:14:40.400419 15573 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:40.771392 15749 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:41.453511 15799 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.627s user 0.005s sys 0.025s
I20260808 20:14:40.236478 15614 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
W20260808 20:14:41.248677 15798 log.cc:924] Time spent T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20260808 20:14:40.692811 15727 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:41.006836 15806 log.cc:924] Time spent T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.218s user 0.001s sys 0.000s
W20260808 20:14:41.262387 15767 log.cc:924] Time spent T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.115s user 0.000s sys 0.000s
I20260808 20:14:41.361145 12036 raft_consensus.cc:1226] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: []
W20260808 20:14:40.561357 15756 log.cc:924] Time spent T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.222s user 0.000s sys 0.003s
W20260808 20:14:40.437021 15615 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.009s user 0.004s sys 0.015s
I20260808 20:14:40.577169 15762 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
W20260808 20:14:40.437181 15107 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.784s user 0.004s sys 0.014s
I20260808 20:14:40.893615 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:40.957064 12047 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.683s user 0.003s sys 0.015s
W20260808 20:14:40.945480 15803 log.cc:924] Time spent T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20260808 20:14:40.771556 15584 raft_consensus.cc:3037] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:40.693064 15591 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:41.339316 12048 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
W20260808 20:14:40.437139 15322 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.842s user 0.002s sys 0.017s
I20260808 20:14:41.369629 13080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53290 (request call id 561) took 2870 ms. Trace:
W20260808 20:14:41.417564 12142 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.675s user 0.020s sys 0.025s
I20260808 20:14:41.420157 12206 leader_election.cc:302] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
W20260808 20:14:41.417949 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 2.1301e+06us
I20260808 20:14:40.697679 13071 rpcz_store.cc:276] 0808 20:14:38.497607 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:38.501270 (+ 3663us) service_pool.cc:227] Handling call
0808 20:14:39.896229 (+1394959us) spinlock_profiling.cc:243] Waited 4.03 ms on lock 0x7b5800005198. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:39.896453 (+ 224us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:39.905670 (+ 9217us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:39.905704 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:39.907679 (+ 1975us) log.cc:842] Serialized 548 byte log entry
0808 20:14:40.106980 (+199301us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:40.109475 (+ 2495us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:40.128596 (+ 19121us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:40.474109 (+345513us) raft_consensus.cc:1671] finished
0808 20:14:40.474132 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:40.474448 (+ 316us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9267456,"thread_start_us":191676,"threads_started":1,"wal-append.queue_time_us":508949}
I20260808 20:14:40.609153 12144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 331) took 2549 ms. Trace:
I20260808 20:14:40.736023 15624 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:40.735281 15735 raft_consensus.cc:2726] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:40.571739 15666 raft_consensus.cc:482] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:40.203366 15635 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:41.418056 12105 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.003s sys 0.013s
W20260808 20:14:41.418643 12249 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.721s user 0.012s sys 0.010s
W20260808 20:14:41.418021 13016 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.842s user 0.006s sys 0.017s
W20260808 20:14:41.425022 12103 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.689s user 0.010s sys 0.004s
W20260808 20:14:40.846334 12250 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.567s user 0.000s sys 0.018s
I20260808 20:14:40.247691 12101 leader_election.cc:302] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:41.065239 15696 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.785s user 0.001s sys 0.020s
I20260808 20:14:40.735323 15612 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:40.788439 15743 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:40.442056 15726 raft_consensus.cc:482] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:40.319013 15700 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:41.433197 11872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 659201us
I20260808 20:14:41.433013 12038 rpcz_store.cc:276] 0808 20:14:40.219878 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:40.220550 (+ 672us) service_pool.cc:227] Handling call
0808 20:14:41.334074 (+1113524us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:41.416865 15667 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.657s user 0.012s sys 0.028s
I20260808 20:14:40.693310 15759 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:40.609333 15602 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:41.421578 13073 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.784s user 0.014s sys 0.019s
I20260808 20:14:40.735085 15674 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:40.609477 15713 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:41.416085 15771 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:14:41.429698 13076 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.834s user 0.011s sys 0.023s
I20260808 20:14:40.631888 15578 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:41.437451 15673 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 3.085s user 0.008s sys 0.035s
I20260808 20:14:40.574034 15748 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:41.450879 13078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53296 (request call id 690) took 2645 ms. Trace:
I20260808 20:14:40.609576 15741 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:40.696399 15733 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:41.456182 12137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1667) took 2729 ms. Trace:
I20260808 20:14:40.257952 15096 consensus_queue.cc:1046] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 1.179s
I20260808 20:14:41.134338 15773 consensus_queue.cc:1035] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.365s
W20260808 20:14:41.449647 15638 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 3.065s user 0.014s sys 0.046s
I20260808 20:14:41.358474 15653 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:40.651656 15739 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:41.440936 15736 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:41.454013 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:41.444348 13074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53290 (request call id 570) took 1226 ms. Trace:
W20260808 20:14:41.455991 15766 log.cc:924] Time spent T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.390s user 0.000s sys 0.001s
I20260808 20:14:41.357513 15697 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:41.413416 15646 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:41.449925 15786 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
W20260808 20:14:41.492264 12001 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.025s sys 0.039s
I20260808 20:14:40.733788 15789 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:41.441195 15571 raft_consensus.cc:504] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:41.441341 15746 raft_consensus.cc:504] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:41.760560 12001 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.783s user 0.026s sys 0.041s
I20260808 20:14:41.742478 12140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1673) took 3012 ms. Trace:
I20260808 20:14:41.748363 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1913) took 2484 ms. Trace:
I20260808 20:14:41.716084 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1439) took 3140 ms. Trace:
I20260808 20:14:41.727798 12147 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1671) took 2998 ms. Trace:
W20260808 20:14:41.507089 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 168098us
W20260808 20:14:41.760524 15667 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.018s user 0.016s sys 0.033s
I20260808 20:14:41.462066 15587 raft_consensus.cc:482] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:41.767045 12105 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.915s user 0.009s sys 0.016s
I20260808 20:14:41.777712 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:41.781257 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 1.53353e+06us
W20260808 20:14:41.766170 12103 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.031s user 0.018s sys 0.005s
W20260808 20:14:41.766902 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 1.43993e+06us
W20260808 20:14:41.765842 12142 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.023s user 0.020s sys 0.034s
I20260808 20:14:41.767899 12046 raft_consensus.cc:2454] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:41.760403 13078 rpcz_store.cc:276] 0808 20:14:38.805792 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:38.982301 (+176509us) service_pool.cc:227] Handling call
0808 20:14:38.994712 (+ 12411us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:39.089103 (+ 94391us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:39.089149 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:39.132564 (+ 43415us) log.cc:842] Serialized 26 byte log entry
0808 20:14:40.128191 (+995627us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:40.128228 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:40.128352 (+ 124us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:41.450675 (+1322323us) raft_consensus.cc:1671] finished
0808 20:14:41.450695 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:41.450820 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":66522,"raft.queue_time_us":84091,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":38929,"thread_start_us":872750,"threads_started":1,"wal-append.queue_time_us":1128996}
I20260808 20:14:41.786136 12046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49226 (request call id 504) took 1021 ms. Trace:
W20260808 20:14:41.454959 15799 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s user 0.006s sys 0.025s
I20260808 20:14:41.771891 12036 rpcz_store.cc:276] 0808 20:14:38.562633 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:38.563023 (+ 390us) service_pool.cc:227] Handling call
0808 20:14:38.571049 (+ 8026us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:41.465388 (+2894339us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:41.702549 (+237161us) spinlock_profiling.cc:243] Waited 2.79 s on lock 0x7b54000f5d54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:41.702769 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6416294656}
I20260808 20:14:41.773545 12249 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1910) took 2488 ms. Trace:
W20260808 20:14:41.770825 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 792979us
I20260808 20:14:41.771631 12041 rpcz_store.cc:276] 0808 20:14:39.264506 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:39.264638 (+ 132us) service_pool.cc:227] Handling call
0808 20:14:39.265014 (+ 376us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:39.265351 (+ 337us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:39.265377 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:39.305480 (+ 40103us) log.cc:842] Serialized 26 byte log entry
0808 20:14:40.688686 (+1383206us) raft_consensus.cc:1631] Marking committed up to 1
0808 20:14:40.688724 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:40.688792 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:41.748140 (+1059348us) raft_consensus.cc:1671] finished
0808 20:14:41.748161 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:41.748300 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":78037,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":9199,"thread_start_us":1347041,"threads_started":1,"wal-append.queue_time_us":1742275}
I20260808 20:14:41.778385 15592 raft_consensus.cc:2781] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260808 20:14:41.715564 13072 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 695) took 1608 ms (client timeout 1822 ms). Trace:
I20260808 20:14:41.455067 15777 raft_consensus.cc:482] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
W20260808 20:14:41.779281 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 363421us
I20260808 20:14:41.781325 12000 leader_election.cc:302] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
W20260808 20:14:41.710189 15747 log.cc:924] Time spent T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.173s user 0.000s sys 0.000s
W20260808 20:14:41.742691 15824 log.cc:924] Time spent T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.246s user 0.000s sys 0.000s
I20260808 20:14:41.714486 15833 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:41.884583 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:41.786715 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 335) took 2856 ms. Trace:
I20260808 20:14:41.891922 12045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 493) took 1665 ms. Trace:
I20260808 20:14:41.761210 12137 rpcz_store.cc:276] 0808 20:14:38.726747 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:38.726893 (+ 146us) service_pool.cc:227] Handling call
0808 20:14:38.727211 (+ 318us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:38.727871 (+ 660us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:38.727898 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:38.807553 (+ 79655us) log.cc:842] Serialized 26 byte log entry
0808 20:14:39.543284 (+735731us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:40.697496 (+1154212us) log.cc:842] Serialized 14 byte log entry
0808 20:14:40.734576 (+ 37080us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:40.734659 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:41.455974 (+721315us) raft_consensus.cc:1671] finished
0808 20:14:41.455996 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:41.456120 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37143,"prepare.queue_time_us":1206268,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":7549,"raft.queue_time_us":516845,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":18509,"thread_start_us":1757104,"threads_started":2,"wal-append.queue_time_us":1408097}
W20260808 20:14:41.767395 15696 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.494s user 0.001s sys 0.040s
W20260808 20:14:41.494064 15107 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.841s user 0.004s sys 0.029s
I20260808 20:14:41.765516 12048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49226 (request call id 496) took 1371 ms. Trace:
W20260808 20:14:41.765522 12250 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.501s user 0.005s sys 0.045s
W20260808 20:14:41.930822 13078 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53290 (request call id 571) took 1717 ms (client timeout 1883 ms). Trace:
W20260808 20:14:41.932866 13078 rpcz_store.cc:269] 0808 20:14:40.213858 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:41.793192 (+1579334us) service_pool.cc:227] Handling call
0808 20:14:41.930752 (+137560us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:41.891198 12142 rpcz_store.cc:276] 0808 20:14:38.929826 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:39.505469 (+575643us) service_pool.cc:227] Handling call
0808 20:14:39.526694 (+ 21225us) raft_consensus.cc:1457] Updating replica for 3 ops
0808 20:14:39.561588 (+ 34894us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:39.561624 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0808 20:14:39.570051 (+ 8427us) log.cc:842] Serialized 78 byte log entry
0808 20:14:40.585432 (+1015381us) raft_consensus.cc:1631] Marking committed up to 8
0808 20:14:41.785974 (+1200542us) log.cc:842] Serialized 14 byte log entry
0808 20:14:41.786421 (+ 447us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:41.786487 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:41.786525 (+ 38us) raft_consensus.cc:1671] finished
0808 20:14:41.786535 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:41.786662 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38260,"prepare.queue_time_us":255097,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":33668,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":16791,"thread_start_us":2055594,"threads_started":2,"wal-append.queue_time_us":1238585}
I20260808 20:14:41.931768 12048 rpcz_store.cc:276] 0808 20:14:40.411801 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:40.411997 (+ 196us) service_pool.cc:227] Handling call
0808 20:14:41.765452 (+1353455us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:41.892352 12045 rpcz_store.cc:276] 0808 20:14:40.226645 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:40.227472 (+ 827us) service_pool.cc:227] Handling call
0808 20:14:40.364856 (+137384us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:41.786004 (+1421148us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:41.786034 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:41.786071 (+ 37us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:41.786098 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:41.891664 (+105566us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b5400000d54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:41.891719 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:41.891857 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3264127360}
I20260808 20:14:41.937105 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 338) took 2867 ms. Trace:
I20260808 20:14:41.937546 12146 rpcz_store.cc:276] 0808 20:14:39.069875 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:39.646136 (+576261us) service_pool.cc:227] Handling call
0808 20:14:39.673549 (+ 27413us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:39.673745 (+ 196us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 20:14:41.438336 (+1764591us) log.cc:842] Serialized 14 byte log entry
0808 20:14:41.936749 (+498413us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:41.936801 (+ 52us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:41.936845 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:41.936927 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:41.937051 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1099905,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"thread_start_us":2179296,"threads_started":2,"wal-append.queue_time_us":498407}
I20260808 20:14:41.940945 12243 raft_consensus.cc:1226] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:41.944705 12150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1677) took 3095 ms. Trace:
W20260808 20:14:41.791221 15815 log.cc:924] Time spent T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.109s user 0.001s sys 0.000s
W20260808 20:14:41.836601 15675 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.234s user 0.007s sys 0.053s
I20260808 20:14:41.766263 13080 rpcz_store.cc:276] 0808 20:14:38.498943 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:38.499070 (+ 127us) service_pool.cc:227] Handling call
0808 20:14:39.240872 (+741802us) spinlock_profiling.cc:243] Waited 4.66 ms on lock 0x7b5800005198. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:39.250375 (+ 9503us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:39.312806 (+ 62431us) raft_consensus.cc:1520] Early marking committed up to index 5
0808 20:14:40.130636 (+817830us) log.cc:842] Serialized 14 byte log entry
0808 20:14:41.335388 (+1204752us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:41.335436 (+ 48us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:41.335478 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:41.369381 (+ 33903us) spinlock_profiling.cc:243] Waited 62.2 ms on lock 0x7b5400524ed4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:41.369430 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:41.369566 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3554,"prepare.queue_time_us":1262975,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":3932,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":22632,"spinlock_wait_cycles":153768960,"thread_start_us":2000858,"threads_started":2,"wal-append.queue_time_us":1583964}
W20260808 20:14:41.494246 15615 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.066s user 0.005s sys 0.031s
I20260808 20:14:41.455235 15769 raft_consensus.cc:482] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:41.790254 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:41.794216 15667 raft_consensus.cc:2781] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:41.454959 15749 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:41.494016 15142 raft_consensus.cc:482] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:41.770571 12140 rpcz_store.cc:276] 0808 20:14:38.729859 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:38.729963 (+ 104us) service_pool.cc:227] Handling call
0808 20:14:38.803723 (+ 73760us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:38.840588 (+ 36865us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:38.840617 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:38.901230 (+ 60613us) log.cc:842] Serialized 26 byte log entry
0808 20:14:40.023781 (+1122551us) raft_consensus.cc:1631] Marking committed up to 1
0808 20:14:40.023825 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:40.023904 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:40.549927 (+526023us) spinlock_profiling.cc:243] Waited 8.15 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:41.727119 (+1177192us) raft_consensus.cc:1671] finished
0808 20:14:41.727140 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:41.742409 (+ 15269us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6565,"raft.queue_time_us":107803,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":13551,"spinlock_wait_cycles":18745600,"thread_start_us":1049836,"threads_started":1,"wal-append.queue_time_us":1354446}
I20260808 20:14:41.682292 15778 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:41.771889 12147 rpcz_store.cc:276] 0808 20:14:38.729234 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:38.729344 (+ 110us) service_pool.cc:227] Handling call
0808 20:14:38.806830 (+ 77486us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 20:14:39.287511 (+480681us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:39.287538 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 20:14:39.316069 (+ 28531us) log.cc:842] Serialized 52 byte log entry
0808 20:14:41.339488 (+2023419us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:41.421835 (+ 82347us) log.cc:842] Serialized 14 byte log entry
0808 20:14:41.422030 (+ 195us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:41.422107 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:41.727600 (+305493us) raft_consensus.cc:1671] finished
0808 20:14:41.727620 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:41.727734 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10114,"prepare.queue_time_us":82193,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":140956,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":35463,"thread_start_us":2388513,"threads_started":2,"wal-append.queue_time_us":1730944}
I20260808 20:14:41.762467 15771 raft_consensus.cc:3037] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:41.495632 13075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53296 (request call id 693) took 1407 ms. Trace:
W20260808 20:14:42.014963 12207 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.008s sys 0.010s
W20260808 20:14:42.015307 12207 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.557s user 0.008s sys 0.010s
W20260808 20:14:42.015545 12207 connection.cc:232] RPC call timeout handler was delayed by -1.2156832760665566s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1665s for a response
I20260808 20:14:42.016197 12207 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:42.016919 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:41.945240 15832 raft_consensus.cc:482] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
W20260808 20:14:41.493525 12047 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.220s user 0.006s sys 0.023s
I20260808 20:14:41.768765 15666 raft_consensus.cc:504] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:41.793179 12249 rpcz_store.cc:276] 0808 20:14:39.285258 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:39.316344 (+ 31086us) service_pool.cc:227] Handling call
0808 20:14:39.340555 (+ 24211us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:39.340759 (+ 204us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 20:14:40.696968 (+1356209us) log.cc:842] Serialized 14 byte log entry
0808 20:14:41.773193 (+1076225us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:41.773244 (+ 51us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:41.773285 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:41.773379 (+ 94us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:41.773491 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":597119,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":6527,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":8493,"thread_start_us":2410141,"threads_started":2,"wal-append.queue_time_us":752814}
I20260808 20:14:41.847687 12206 leader_election.cc:302] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:42.018898 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 171296us
I20260808 20:14:42.019606 12206 leader_election.cc:302] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:41.493192 15776 raft_consensus.cc:482] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:42.021234 15776 raft_consensus.cc:504] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:42.023440 15776 leader_election.cc:288] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:42.023993 15776 raft_consensus.cc:2781] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:42.024324 15776 raft_consensus.cc:482] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:42.024659 15776 raft_consensus.cc:3037] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:41.533762 12141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 328) took 3504 ms. Trace:
I20260808 20:14:41.835565 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:42.031159 12036 raft_consensus.cc:2454] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:42.031637 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49226 (request call id 497) took 1611 ms. Trace:
I20260808 20:14:42.031924 12036 rpcz_store.cc:276] 0808 20:14:40.420330 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:41.789824 (+1369494us) service_pool.cc:227] Handling call
0808 20:14:42.031579 (+241755us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:41.462193 15614 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:41.790941 15816 log.cc:924] Time spent T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
I20260808 20:14:41.896196 15743 consensus_queue.cc:1035] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.115s
I20260808 20:14:41.945034 12150 rpcz_store.cc:276] 0808 20:14:38.849460 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:38.940474 (+ 91014us) service_pool.cc:227] Handling call
0808 20:14:39.614915 (+674441us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:39.644545 (+ 29630us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:39.644575 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:39.791998 (+147423us) log.cc:842] Serialized 26 byte log entry
0808 20:14:41.331306 (+1539308us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:41.331408 (+ 102us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:41.331489 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:41.724925 (+393436us) raft_consensus.cc:1671] finished
0808 20:14:41.724949 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:41.896979 (+172030us) spinlock_profiling.cc:243] Waited 642 ms on lock 0x7b54004259d0. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d9a09 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:41.944637 (+ 47658us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29255,"raft.queue_time_us":29545,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":6931,"spinlock_wait_cycles":1475608832,"thread_start_us":1512005,"threads_started":1,"wal-append.queue_time_us":929843}
I20260808 20:14:41.726749 15837 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:41.894753 15322 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.300s user 0.014s sys 0.034s
I20260808 20:14:41.774963 15591 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:41.843209 15700 raft_consensus.cc:482] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:41.848989 15697 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:41.953379 15830 log.cc:924] Time spent T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045: Append to log took a long time: real 0.231s user 0.000s sys 0.000s
I20260808 20:14:41.956198 15769 raft_consensus.cc:504] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:41.833463 15789 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:41.958262 15571 leader_election.cc:288] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:41.791587 15733 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:41.847415 15592 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:41.961355 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49226 (request call id 499) took 1511 ms. Trace:
I20260808 20:14:41.962266 15748 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:41.457747 15758 raft_consensus.cc:504] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:41.963037 15699 consensus_queue.cc:1035] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:41.768397 12144 rpcz_store.cc:276] 0808 20:14:38.059861 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:38.073117 (+ 13256us) service_pool.cc:227] Handling call
0808 20:14:39.222317 (+1149200us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:39.222590 (+ 273us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 20:14:39.223334 (+ 744us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:39.223388 (+ 54us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 20:14:39.224409 (+ 1021us) log.cc:842] Serialized 52 byte log entry
0808 20:14:40.193313 (+968904us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:40.198412 (+ 5099us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:40.212685 (+ 14273us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:40.559345 (+346660us) raft_consensus.cc:1671] finished
0808 20:14:40.562231 (+ 2886us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:40.609052 (+ 46821us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10593,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":188,"spinlock_wait_cycles":23105280,"thread_start_us":901324,"threads_started":1,"wal-append.queue_time_us":1109527,"wal-append.run_cpu_time_us":3375,"wal-append.run_wall_time_us":1269128}
W20260808 20:14:41.875666 13072 rpcz_store.cc:269] 0808 20:14:40.107097 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:40.109710 (+ 2613us) service_pool.cc:227] Handling call
0808 20:14:40.893401 (+783691us) spinlock_profiling.cc:243] Waited 5.3 ms on lock 0x7b5800005198. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:41.708258 (+814857us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12183808}
I20260808 20:14:41.964445 15674 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:41.850134 15746 leader_election.cc:288] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:41.965728 13079 raft_consensus.cc:1226] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260808 20:14:41.760694 13074 rpcz_store.cc:276] 0808 20:14:40.213568 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:40.236561 (+ 22993us) service_pool.cc:227] Handling call
0808 20:14:41.439767 (+1203206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:41.850267 15791 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:41.967128 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:41.946987 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 154197us
W20260808 20:14:41.658411 15779 log.cc:924] Time spent T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.164s user 0.005s sys 0.000s
I20260808 20:14:41.976289 12048 raft_consensus.cc:1226] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 6. Current term is 7. Ops: []
I20260808 20:14:41.848193 13076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53290 (request call id 560) took 3350 ms. Trace:
I20260808 20:14:41.836937 15828 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
W20260808 20:14:41.998353 12251 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.006s sys 0.008s
I20260808 20:14:42.001061 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:42.005280 11994 connection.cc:232] RPC call timeout handler was delayed by -2.8045779778622091s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2155s for a response
I20260808 20:14:41.781116 15736 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:41.773063 15635 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:41.850244 15741 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:42.009655 13077 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.035s sys 0.071s
I20260808 20:14:41.964386 15739 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:41.811566 15096 consensus_queue.cc:1035] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.162s
I20260808 20:14:41.774753 15624 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:41.930907 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:41.508567 15762 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:42.013854 12247 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.684s user 0.007s sys 0.011s
I20260808 20:14:42.014011 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:41.775265 15612 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:42.025075 15855 raft_consensus.cc:2726] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:41.786478 12046 rpcz_store.cc:276] 0808 20:14:40.765247 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:41.427648 (+662401us) service_pool.cc:227] Handling call
0808 20:14:41.786031 (+358383us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:41.767340 15786 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:41.959977 15653 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:41.895080 15835 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:41.849098 15727 consensus_queue.cc:1046] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.148s
I20260808 20:14:41.945461 12043 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1911) took 2788 ms. Trace:
I20260808 20:14:41.975261 15667 raft_consensus.cc:482] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:42.046064 12036 raft_consensus.cc:3032] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:41.790366 15587 raft_consensus.cc:504] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:41.787431 15726 raft_consensus.cc:504] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:42.048599 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 267355us
W20260808 20:14:42.050896 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 241396us
I20260808 20:14:42.054162 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:42.060078 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 229436us
I20260808 20:14:41.833457 15834 consensus_queue.cc:1035] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.214s
I20260808 20:14:42.015522 15713 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:42.063398 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:42.064693 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 491) took 1845 ms. Trace:
I20260808 20:14:41.773253 15759 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:42.078389 15690 raft_consensus.cc:2781] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:42.045531 15849 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:42.011093 15142 raft_consensus.cc:504] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:41.888830 15777 raft_consensus.cc:504] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:41.947330 15833 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:42.502091 12036 raft_consensus.cc:729] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:42.502544 13071 raft_consensus.cc:2454] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:42.228766 15596 raft_consensus.cc:2781] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:42.503738 12048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 500) took 2018 ms. Trace:
I20260808 20:14:42.505803 12254 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:42.505954 13080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57330 (request call id 479) took 1058 ms. Trace:
W20260808 20:14:42.506639 12251 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.045s user 0.009s sys 0.021s
W20260808 20:14:42.506458 11994 connection.cc:232] RPC call timeout handler was delayed by -2.7307779779657722s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2237s for a response
W20260808 20:14:42.507618 11994 connection.cc:232] RPC call timeout handler was delayed by -2.711796801770106s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1583s for a response
I20260808 20:14:42.035609 15778 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:42.036450 15832 raft_consensus.cc:504] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:42.059204 15837 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:42.506394 13080 rpcz_store.cc:276] 0808 20:14:41.447942 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:42.049107 (+601165us) service_pool.cc:227] Handling call
0808 20:14:42.505895 (+456788us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:42.512756 12043 rpcz_store.cc:276] 0808 20:14:39.157370 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:39.255294 (+ 97924us) service_pool.cc:227] Handling call
0808 20:14:39.280161 (+ 24867us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:39.286315 (+ 6154us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:39.286347 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:39.309607 (+ 23260us) log.cc:842] Serialized 26 byte log entry
0808 20:14:40.126007 (+816400us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:40.126047 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:40.243869 (+117822us) spinlock_profiling.cc:243] Waited 5.82 ms on lock 0x7b540012bbd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:40.243920 (+ 51us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:41.894656 (+1650736us) raft_consensus.cc:1671] finished
0808 20:14:41.894681 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:41.945392 (+ 50711us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":73740,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":117,"spinlock_wait_cycles":13389056,"thread_start_us":785704,"threads_started":1,"wal-append.queue_time_us":1544832}
I20260808 20:14:42.386313 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1915) took 3101 ms. Trace:
W20260808 20:14:42.190734 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 174623us
I20260808 20:14:42.503801 12040 rpcz_store.cc:276] 0808 20:14:40.219602 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:40.219705 (+ 103us) service_pool.cc:227] Handling call
0808 20:14:40.260684 (+ 40979us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:41.792803 (+1532119us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:41.792833 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:41.792868 (+ 35us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:14:41.792931 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:42.064471 (+271540us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b540004e7d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:42.064515 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:42.064634 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3523152256}
I20260808 20:14:42.548977 12039 rpcz_store.cc:276] 0808 20:14:39.285288 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:39.285419 (+ 131us) service_pool.cc:227] Handling call
0808 20:14:39.309361 (+ 23942us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:40.047169 (+737808us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:40.047199 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:40.047633 (+ 434us) log.cc:842] Serialized 26 byte log entry
0808 20:14:41.400656 (+1353023us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:14:41.400698 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:41.400775 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:42.386074 (+985299us) raft_consensus.cc:1671] finished
0808 20:14:42.386096 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:42.386250 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":45613,"raft.run_cpu_time_us":68,"raft.run_wall_time_us":68,"thread_start_us":1962387,"threads_started":2,"wal-append.queue_time_us":1412099}
I20260808 20:14:42.015188 13075 rpcz_store.cc:276] 0808 20:14:40.087516 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:40.090676 (+ 3160us) service_pool.cc:227] Handling call
0808 20:14:40.235811 (+145135us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:41.455550 (+1219739us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:41.495024 (+ 39474us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:42.517486 13077 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.086s user 0.044s sys 0.085s
I20260808 20:14:42.517946 12245 raft_consensus.cc:2454] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:42.505496 12048 rpcz_store.cc:276] 0808 20:14:40.486255 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:41.934857 (+1448602us) service_pool.cc:227] Handling call
0808 20:14:41.976115 (+ 41258us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:42.503498 (+527383us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:42.503686 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:42.524996 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:42.525851 12247 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.196s user 0.012s sys 0.013s
I20260808 20:14:42.502844 15667 raft_consensus.cc:3037] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:42.550715 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:42.714881 13080 raft_consensus.cc:2379] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 5: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
W20260808 20:14:42.667991 12244 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.724s user 0.006s sys 0.007s
W20260808 20:14:42.715372 12244 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.771s user 0.007s sys 0.007s
W20260808 20:14:42.633424 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 46244.5us
W20260808 20:14:42.177820 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 212305us
I20260808 20:14:42.337776 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:42.562491 12047 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.006s sys 0.015s
I20260808 20:14:42.296298 15602 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:42.067332 15650 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:42.136884 12241 raft_consensus.cc:1226] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 8. Current term is 9. Ops: []
I20260808 20:14:42.067703 15700 raft_consensus.cc:504] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:42.504942 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:42.504336 15587 leader_election.cc:288] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:42.561888 12041 rpcz_store.cc:276] 0808 20:14:40.450258 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:41.790018 (+1339760us) service_pool.cc:227] Handling call
0808 20:14:41.961282 (+171264us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:42.608623 12038 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
W20260808 20:14:42.716993 12047 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.699s user 0.007s sys 0.015s
I20260808 20:14:42.605751 12207 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:42.714747 12045 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.737s user 0.009s sys 0.003s
W20260808 20:14:42.738946 12045 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.762s user 0.010s sys 0.003s
W20260808 20:14:42.624396 15821 log.cc:924] Time spent T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.880s user 0.003s sys 0.006s
I20260808 20:14:42.587067 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:42.605484 13078 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:42.720817 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 506) took 1743 ms. Trace:
I20260808 20:14:42.278455 15646 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:42.726112 12001 connection.cc:232] RPC call timeout handler was delayed by -1.0257068169303238s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16570000000000001s for a response
I20260808 20:14:42.503544 15833 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:42.703019 13119 heartbeater.cc:644] Failed to heartbeat to 127.10.24.62:36417 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.10.24.62:36417 timed out after 15.000s (SENT)
W20260808 20:14:42.703141 15851 log.cc:924] Time spent T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.407s user 0.000s sys 0.000s
I20260808 20:14:42.385365 15858 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:42.655756 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:42.740216 12047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 495) took 2510 ms. Trace:
I20260808 20:14:42.741123 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:42.848017 12041 raft_consensus.cc:2454] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:42.719175 12000 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:42.849319 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 130238us
I20260808 20:14:42.851166 12038 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6.
I20260808 20:14:42.505514 15690 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:42.586948 15733 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:42.747000 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:42.821075 12043 raft_consensus.cc:2396] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260808 20:14:42.705570 15142 leader_election.cc:288] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:42.698693 15826 raft_consensus.cc:3032] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 20:14:42.818660 12045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 501) took 2289 ms. Trace:
I20260808 20:14:42.872766 12045 rpcz_store.cc:276] 0808 20:14:40.529043 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:41.947831 (+1418788us) service_pool.cc:227] Handling call
0808 20:14:41.976842 (+ 29011us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:42.739385 (+762543us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:42.739418 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:42.739468 (+ 50us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:42.818363 (+ 78895us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:42.818451 (+ 88us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:42.818594 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":154,"raft.queue_time_us":201286,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":111,"thread_start_us":761960,"threads_started":1}
I20260808 20:14:42.820053 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:42.713828 12254 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
W20260808 20:14:42.823005 13078 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53290 (request call id 572) took 2587 ms (client timeout 2121 ms). Trace:
W20260808 20:14:42.875245 13078 rpcz_store.cc:269] 0808 20:14:40.236007 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:41.933757 (+1697750us) service_pool.cc:227] Handling call
0808 20:14:42.822927 (+889170us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:42.823596 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 217937us
I20260808 20:14:42.875674 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:42.876336 12254 raft_consensus.cc:2454] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:42.075628 15853 consensus_queue.cc:1046] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.660s
I20260808 20:14:42.822929 12039 rpcz_store.cc:276] 0808 20:14:40.977196 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:42.551633 (+1574437us) service_pool.cc:227] Handling call
0808 20:14:42.717794 (+166161us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:42.720486 (+ 2692us) raft_consensus.cc:1520] Early marking committed up to index 5
0808 20:14:42.720516 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:42.720555 (+ 39us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:42.720594 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:42.720660 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:42.720764 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2326,"raft.queue_time_us":2414,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":4902}
I20260808 20:14:42.878113 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:42.636018 15612 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:42.297086 15578 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:42.550526 15739 consensus_queue.cc:1046] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.178s
I20260808 20:14:42.629155 15749 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:42.832352 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:42.513899 15592 raft_consensus.cc:3037] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:42.636731 15635 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:42.627972 15736 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:42.720989 15322 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.011s sys 0.042s
I20260808 20:14:42.626605 15624 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:42.855046 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:42.531960 13071 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53290 (request call id 573) took 2265 ms (client timeout 1657 ms). Trace:
I20260808 20:14:42.903473 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:42.904649 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:42.842799 15614 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:42.905341 12045 raft_consensus.cc:2454] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:42.845165 15587 raft_consensus.cc:2726] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 20:14:42.860484 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:42.870985 15826 raft_consensus.cc:729] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:42.866444 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:42.898061 15322 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.812s user 0.014s sys 0.046s
I20260808 20:14:42.628401 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:42.514091 15849 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:42.742213 15650 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:42.532810 15596 raft_consensus.cc:686] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 5 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:42.870787 15733 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:42.526973 15835 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:42.588081 15828 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:42.896466 12048 raft_consensus.cc:2396] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260808 20:14:42.874766 13080 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:14:42.989624 13080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 697) took 1196 ms. Trace:
I20260808 20:14:42.910769 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:42.867811 15690 raft_consensus.cc:3037] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:42.951521 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:42.635335 15713 consensus_queue.cc:1046] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.247s
I20260808 20:14:42.527359 15571 consensus_queue.cc:1035] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.295s
I20260808 20:14:43.001189 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:42.916630 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:42.921660 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:42.846731 12047 rpcz_store.cc:276] 0808 20:14:40.229724 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:40.231218 (+ 1494us) service_pool.cc:227] Handling call
0808 20:14:40.263543 (+ 32325us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:40.263734 (+ 191us) raft_consensus.cc:1520] Early marking committed up to index 7
0808 20:14:40.273624 (+ 9890us) log.cc:842] Serialized 14 byte log entry
0808 20:14:42.739880 (+2466256us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:42.739932 (+ 52us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:14:42.739976 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:42.740037 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:42.740158 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28402,"prepare.queue_time_us":9733,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":12454,"raft.queue_time_us":173487,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":40724,"thread_start_us":2465880,"threads_started":2,"wal-append.queue_time_us":1518743}
I20260808 20:14:43.002789 12254 raft_consensus.cc:2396] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260808 20:14:42.865674 15852 raft_consensus.cc:2781] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 20:14:42.924779 12041 raft_consensus.cc:2454] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:42.951911 12245 raft_consensus.cc:2379] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 8: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:42.953312 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:42.954281 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 446198us
I20260808 20:14:42.954753 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:42.969139 12206 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.936s user 0.031s sys 0.073s
W20260808 20:14:43.004252 12206 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.971s user 0.031s sys 0.073s
I20260808 20:14:42.869429 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:42.569949 15638 consensus_queue.cc:1035] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.327s
I20260808 20:14:42.863071 15858 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:42.720871 15862 consensus_queue.cc:1046] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.204s
I20260808 20:14:42.895277 15870 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:42.885195 13072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57330 (request call id 480) took 1432 ms. Trace:
I20260808 20:14:43.015925 13072 rpcz_store.cc:276] 0808 20:14:41.453030 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:42.504647 (+1051617us) service_pool.cc:227] Handling call
0808 20:14:42.871617 (+366970us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:43.017211 15828 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:43.017460 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:42.976255 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 327012us
I20260808 20:14:43.018402 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:42.924505 15877 log.cc:924] Time spent T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.201s user 0.000s sys 0.000s
I20260808 20:14:42.626029 13073 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53290 (request call id 568) took 3326 ms. Trace:
W20260808 20:14:42.980420 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 255244us
I20260808 20:14:42.679016 15791 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:42.913730 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:42.587493 15867 consensus_queue.cc:1035] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.434s
I20260808 20:14:42.883241 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:42.729324 15591 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:42.989946 13080 rpcz_store.cc:276] 0808 20:14:41.793492 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:42.724119 (+930627us) service_pool.cc:227] Handling call
0808 20:14:42.989559 (+265440us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:42.902969 13071 rpcz_store.cc:269] 0808 20:14:40.255476 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:41.811044 (+1555568us) service_pool.cc:227] Handling call
0808 20:14:42.520370 (+709326us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:42.880869 15584 consensus_meta.cc:287] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 1.373s user 0.008s sys 0.002s
I20260808 20:14:42.924490 15826 consensus_queue.cc:260] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:42.998838 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:43.044651 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:43.044766 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:43.014582 15759 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:43.019547 13074 raft_consensus.cc:2454] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:43.045838 13074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 703) took 1034 ms. Trace:
I20260808 20:14:43.046046 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:43.046208 13074 rpcz_store.cc:276] 0808 20:14:42.011947 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:42.713797 (+701850us) service_pool.cc:227] Handling call
0808 20:14:43.045777 (+331980us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:43.046691 12137 raft_consensus.cc:3037] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:43.048386 15735 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:43.043663 15786 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:43.049928 15735 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:43.056159 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:43.012756 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:43.013180 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:43.014585 15758 leader_election.cc:288] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:43.014721 15789 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:43.014961 15587 raft_consensus.cc:482] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:43.015492 15733 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:43.067973 15587 raft_consensus.cc:504] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:43.062254 15789 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:43.069557 13077 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53296 (request call id 694) took 2976 ms. Trace:
I20260808 20:14:43.069936 15587 leader_election.cc:288] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:43.070109 15789 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:42.862601 15833 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:43.070684 15778 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:43.070866 15833 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:43.071388 15650 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:43.049571 15739 consensus_queue.cc:1225] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer 78b9263889cd4f1582578a3aa4f644d9 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260808 20:14:43.060365 15853 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:43.019804 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:43.073225 15624 raft_consensus.cc:3032] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:43.073599 15624 raft_consensus.cc:729] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:43.074333 15624 consensus_queue.cc:260] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:43.075294 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:43.075461 15624 raft_consensus.cc:3037] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
W20260808 20:14:43.023361 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 304523us
I20260808 20:14:43.019019 12041 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:43.079916 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:43.067879 15726 leader_election.cc:288] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:43.080957 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:43.081390 15885 log.cc:924] Time spent T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
I20260808 20:14:43.081584 12041 raft_consensus.cc:2454] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:43.082594 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:43.083587 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:43.007177 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:43.040912 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:43.041947 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:43.087190 12152 raft_consensus.cc:3037] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:43.016111 15837 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:43.042418 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:43.043859 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 166748us
I20260808 20:14:43.045794 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:43.088060 12048 raft_consensus.cc:2454] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
W20260808 20:14:43.005329 11994 connection.cc:232] RPC call timeout handler was delayed by -2.35700272093527s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2049s for a response
I20260808 20:14:43.015552 15142 raft_consensus.cc:482] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:43.049365 12147 raft_consensus.cc:1226] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260808 20:14:43.006036 13078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 702) took 1007 ms. Trace:
I20260808 20:14:43.015425 15884 raft_consensus.cc:482] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:43.057307 12043 raft_consensus.cc:2454] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:43.058899 12137 raft_consensus.cc:2454] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:43.034080 12241 raft_consensus.cc:2396] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260808 20:14:43.060591 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:43.043716 15769 leader_election.cc:288] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:43.064954 15857 log.cc:924] Time spent T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.341s user 0.000s sys 0.002s
I20260808 20:14:43.065302 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:43.050602 15700 leader_election.cc:288] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:43.066205 15880 log.cc:924] Time spent T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20260808 20:14:43.067199 15735 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:43.055864 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:43.049346 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:43.056140 15762 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:43.028514 13014 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:43.015405 15777 leader_election.cc:288] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:43.073195 12047 raft_consensus.cc:2454] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:43.085197 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:43.085633 15833 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:43.054500 15852 raft_consensus.cc:686] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:43.026460 15870 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:43.072347 15853 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:43.053365 15596 consensus_queue.cc:237] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:43.069908 13077 rpcz_store.cc:276] 0808 20:14:40.093551 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:40.093778 (+ 227us) service_pool.cc:227] Handling call
0808 20:14:40.094103 (+ 325us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:40.255744 (+161641us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 20:14:41.431238 (+1175494us) log.cc:842] Serialized 14 byte log entry
0808 20:14:42.713855 (+1282617us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:42.713916 (+ 61us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:42.713960 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:43.069345 (+355385us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b5400652054. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:43.069387 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:43.069507 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":320616,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":28,"raft.queue_time_us":92,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":8563,"spinlock_wait_cycles":395051008,"thread_start_us":2447659,"threads_started":2,"wal-append.queue_time_us":796668}
I20260808 20:14:43.323892 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:43.349450 11994 connection.cc:232] RPC call timeout handler was delayed by -2.1592779778875411s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2872s for a response
I20260808 20:14:43.349536 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:43.156153 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:43.327445 12245 raft_consensus.cc:2454] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:43.090159 12039 raft_consensus.cc:2396] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260808 20:14:43.351132 13078 rpcz_store.cc:276] 0808 20:14:41.998965 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:42.894976 (+896011us) service_pool.cc:227] Handling call
0808 20:14:43.005978 (+111002us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:43.131014 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:43.349870 11994 connection.cc:232] RPC call timeout handler was delayed by -1.7839779779314995s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3289s for a response
I20260808 20:14:42.033996 12141 rpcz_store.cc:276] 0808 20:14:38.029604 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:38.059437 (+ 29833us) service_pool.cc:227] Handling call
0808 20:14:39.122066 (+1062629us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:39.122441 (+ 375us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:39.241390 (+118949us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:39.255501 (+ 14111us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:39.331124 (+ 75623us) log.cc:842] Serialized 26 byte log entry
0808 20:14:41.441971 (+2110847us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:41.454709 (+ 12738us) log.cc:842] Serialized 14 byte log entry
0808 20:14:41.454925 (+ 216us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:41.455025 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:41.461531 (+ 6506us) raft_consensus.cc:1671] finished
0808 20:14:41.490524 (+ 28993us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:41.509192 (+ 18668us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":81711,"prepare.queue_time_us":8499,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":130231,"raft.run_cpu_time_us":377,"raft.run_wall_time_us":94898,"spinlock_wait_cycles":23360896,"thread_start_us":2060687,"threads_started":1,"wal-append.queue_time_us":1772410}
I20260808 20:14:42.506337 13076 rpcz_store.cc:276] 0808 20:14:38.498208 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:38.498353 (+ 145us) service_pool.cc:227] Handling call
0808 20:14:39.593718 (+1095365us) spinlock_profiling.cc:243] Waited 7.56 ms on lock 0x7b5800005198. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:39.594066 (+ 348us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:39.594668 (+ 602us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 20:14:39.594742 (+ 74us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:39.595359 (+ 617us) log.cc:842] Serialized 26 byte log entry
0808 20:14:41.772935 (+2177576us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:41.784321 (+ 11386us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:41.793228 (+ 8907us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:41.797148 (+ 3920us) raft_consensus.cc:1671] finished
0808 20:14:41.836313 (+ 39165us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:41.848125 (+ 11812us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7,"raft.queue_time_us":351,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":117,"spinlock_wait_cycles":17386240,"thread_start_us":2165066,"threads_started":1,"wal-append.queue_time_us":1087883,"wal-append.run_cpu_time_us":1696,"wal-append.run_wall_time_us":1060460}
W20260808 20:14:43.356557 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 328055us
I20260808 20:14:43.064841 15858 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:43.362985 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:43.363755 13077 raft_consensus.cc:2379] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 6: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
W20260808 20:14:43.363749 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 100451us
W20260808 20:14:43.186954 15771 consensus_meta.cc:287] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 1.140s user 0.000s sys 0.014s
I20260808 20:14:43.360090 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:43.397804 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:43.348543 12040 raft_consensus.cc:2454] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
W20260808 20:14:43.362110 11994 connection.cc:232] RPC call timeout handler was delayed by -1.0482389209792018s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2009s for a response
I20260808 20:14:43.352676 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:43.365360 13015 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:43.172420 15835 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:43.391382 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:43.392020 13016 leader_election.cc:334] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.600s (SENT)
I20260808 20:14:43.392182 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:43.351886 12048 raft_consensus.cc:2454] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
W20260808 20:14:43.187969 15869 log.cc:924] Time spent T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.237s user 0.001s sys 0.001s
W20260808 20:14:43.215399 15877 log.cc:924] Time spent T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.133s user 0.001s sys 0.000s
I20260808 20:14:43.394183 13014 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:43.391510 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:43.262187 15776 consensus_meta.cc:287] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 1.213s user 0.007s sys 0.003s
I20260808 20:14:43.309746 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:43.333244 15791 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:43.171339 15849 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:43.175540 15107 consensus_queue.cc:1035] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.266s
I20260808 20:14:43.319543 15592 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:43.398979 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 48540us
I20260808 20:14:43.533353 13079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53290 (request call id 567) took 4729 ms. Trace:
W20260808 20:14:44.113202 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 747880us
I20260808 20:14:44.114058 13015 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:43.028762 13073 rpcz_store.cc:276] 0808 20:14:39.299964 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:39.349336 (+ 49372us) service_pool.cc:227] Handling call
0808 20:14:39.407588 (+ 58252us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:39.635882 (+228294us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:39.635948 (+ 66us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:39.636570 (+ 622us) log.cc:842] Serialized 26 byte log entry
0808 20:14:41.781562 (+2144992us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:41.786375 (+ 4813us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:42.625632 (+839257us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b54004a2c54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:42.625680 (+ 48us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:42.625730 (+ 50us) raft_consensus.cc:1671] finished
0808 20:14:42.625745 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:42.625966 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10616,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":5962,"spinlock_wait_cycles":512253184,"thread_start_us":2125720,"threads_started":1,"wal-append.queue_time_us":1105521,"wal-append.run_cpu_time_us":25954,"wal-append.run_wall_time_us":2047840}
I20260808 20:14:44.114966 15891 raft_consensus.cc:2781] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:44.115383 15891 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:43.415043 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:44.115772 15891 raft_consensus.cc:3037] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:14:43.519591 12038 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49226 (request call id 498) took 3078 ms (client timeout 2113 ms). Trace:
I20260808 20:14:44.113049 13079 rpcz_store.cc:276] 0808 20:14:38.803984 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:38.937381 (+133397us) service_pool.cc:227] Handling call
0808 20:14:38.937695 (+ 314us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:42.561817 (+3624122us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:43.533043 (+971226us) spinlock_profiling.cc:243] Waited 3.01 s on lock 0x7b54001d9a54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:43.533254 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6931856768}
I20260808 20:14:44.117692 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1458) took 1144 ms. Trace:
W20260808 20:14:44.118724 12152 consensus_meta.cc:287] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 1.031s user 0.007s sys 0.026s
I20260808 20:14:43.536074 12036 consensus_queue.cc:260] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:43.320525 13072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 478) took 1885 ms. Trace:
I20260808 20:14:44.118577 12040 rpcz_store.cc:276] 0808 20:14:42.973631 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.084946 (+111315us) service_pool.cc:227] Handling call
0808 20:14:44.117634 (+1032688us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.116385 12047 raft_consensus.cc:2362] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6. Re-sending same reply.
I20260808 20:14:44.128216 12245 raft_consensus.cc:2396] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260808 20:14:44.128612 12047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49226 (request call id 512) took 1131 ms. Trace:
I20260808 20:14:44.112504 12048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 1938) took 1035 ms. Trace:
W20260808 20:14:44.117363 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 723204us
I20260808 20:14:44.129539 12048 rpcz_store.cc:276] 0808 20:14:43.076834 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.155827 (+ 78993us) service_pool.cc:227] Handling call
0808 20:14:44.112429 (+956602us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.131031 12043 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 1941) took 1026 ms. Trace:
I20260808 20:14:44.131381 12043 rpcz_store.cc:276] 0808 20:14:43.105104 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.349107 (+244003us) service_pool.cc:227] Handling call
0808 20:14:44.130969 (+781862us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.112534 12102 leader_election.cc:302] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:44.119550 12036 raft_consensus.cc:3037] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
W20260808 20:14:43.499150 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 107658us
I20260808 20:14:44.133286 12105 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:43.512589 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:44.121287 12137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1398) took 1079 ms. Trace:
I20260808 20:14:44.122288 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:43.512032 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:43.774005 12000 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.003s sys 0.014s
W20260808 20:14:44.118040 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 725999us
I20260808 20:14:44.123847 12152 raft_consensus.cc:2396] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 20:14:44.124166 13015 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:43.054643 15584 raft_consensus.cc:2726] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:44.120934 13080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 696) took 2331 ms. Trace:
I20260808 20:14:43.434949 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.128683 12245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1759) took 1113 ms. Trace:
I20260808 20:14:44.138224 13080 rpcz_store.cc:276] 0808 20:14:41.789423 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.050865 (+1261442us) service_pool.cc:227] Handling call
0808 20:14:44.120848 (+1069983us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.138173 12245 rpcz_store.cc:276] 0808 20:14:43.015094 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.352434 (+337340us) service_pool.cc:227] Handling call
0808 20:14:44.128629 (+776195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.139020 12102 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:44.134306 12142 raft_consensus.cc:2396] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260808 20:14:43.365787 15833 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
W20260808 20:14:44.135157 12000 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.048s user 0.005s sys 0.020s
I20260808 20:14:44.129150 12047 rpcz_store.cc:276] 0808 20:14:42.996961 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.362068 (+365107us) service_pool.cc:227] Handling call
0808 20:14:44.128548 (+766480us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:43.504854 12146 raft_consensus.cc:1226] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260808 20:14:44.134804 12140 raft_consensus.cc:2362] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5. Re-sending same reply.
W20260808 20:14:44.117321 12038 rpcz_store.cc:269] 0808 20:14:40.441406 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:41.775500 (+1334094us) service_pool.cc:227] Handling call
0808 20:14:43.519363 (+1743863us) spinlock_profiling.cc:243] Waited 828 ms on lock 0x7b540046ffd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:43.519489 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1903341440}
I20260808 20:14:44.129740 12244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 420) took 2190 ms. Trace:
I20260808 20:14:44.136476 12254 raft_consensus.cc:2379] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 7: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:44.134032 12137 rpcz_store.cc:276] 0808 20:14:43.041373 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.045807 (+ 4434us) service_pool.cc:227] Handling call
0808 20:14:44.121230 (+1075423us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:43.503671 12251 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1906) took 4246 ms. Trace:
W20260808 20:14:44.140348 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 1.05348e+06us
I20260808 20:14:43.439077 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.140637 12140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1390) took 1107 ms. Trace:
I20260808 20:14:44.140846 12244 rpcz_store.cc:276] 0808 20:14:41.939477 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:41.942794 (+ 3317us) service_pool.cc:227] Handling call
0808 20:14:41.943103 (+ 309us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:41.943488 (+ 385us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:41.943516 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:41.943988 (+ 472us) log.cc:842] Serialized 26 byte log entry
0808 20:14:42.715704 (+771716us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:42.715736 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:42.715798 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:44.129534 (+1413736us) raft_consensus.cc:1671] finished
0808 20:14:44.129553 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:44.129681 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":824308,"raft.queue_time_us":589,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":129,"thread_start_us":771318,"threads_started":1,"wal-append.queue_time_us":941401}
I20260808 20:14:44.142158 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:43.433291 13075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53296 (request call id 698) took 1595 ms. Trace:
I20260808 20:14:43.396049 15653 consensus_queue.cc:1046] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 8.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.752s
I20260808 20:14:44.142992 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:43.366740 15571 raft_consensus.cc:2781] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:43.499001 13071 raft_consensus.cc:2396] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 20:14:43.436839 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:43.640129 15852 consensus_queue.cc:237] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:44.144200 12249 raft_consensus.cc:2454] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:44.142836 13075 rpcz_store.cc:276] 0808 20:14:41.837499 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.156228 (+1318729us) service_pool.cc:227] Handling call
0808 20:14:43.218289 (+ 62061us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:43.360192 (+141903us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:43.362770 (+ 2578us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:43.394671 (+ 31901us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:43.402651 (+ 7980us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:43.433078 (+ 30427us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:43.433217 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:43.359477 15884 raft_consensus.cc:504] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:44.145030 12105 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:44.143846 13071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 699) took 2259 ms. Trace:
I20260808 20:14:44.145560 13071 rpcz_store.cc:276] 0808 20:14:41.884563 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.049088 (+1164525us) service_pool.cc:227] Handling call
0808 20:14:44.143780 (+1094692us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.146509 12254 raft_consensus.cc:2379] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 3: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:44.146241 15833 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:44.146658 15884 leader_election.cc:288] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:43.371457 13074 raft_consensus.cc:2454] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:44.147280 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.147397 13074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 700) took 2217 ms. Trace:
I20260808 20:14:43.433346 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.142827 12243 raft_consensus.cc:2454] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:44.148063 15699 raft_consensus.cc:2781] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:43.397276 15666 leader_election.cc:288] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:44.148391 15571 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:44.139889 15873 raft_consensus.cc:482] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:44.142765 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1392) took 1102 ms. Trace:
W20260808 20:14:43.437640 15667 consensus_meta.cc:287] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.710s user 0.011s sys 0.001s
I20260808 20:14:44.141611 12140 rpcz_store.cc:276] 0808 20:14:43.033549 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.033662 (+ 113us) service_pool.cc:227] Handling call
0808 20:14:43.511870 (+478208us) spinlock_profiling.cc:243] Waited 1.15 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:44.140586 (+628716us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2643840}
I20260808 20:14:44.149600 15571 raft_consensus.cc:3037] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:44.150398 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.150629 15646 raft_consensus.cc:3032] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:44.150997 15646 raft_consensus.cc:729] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:44.151113 12140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1683) took 1089 ms. Trace:
I20260808 20:14:44.148275 15833 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:44.151448 12140 rpcz_store.cc:276] 0808 20:14:43.061957 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.150159 (+1088202us) service_pool.cc:227] Handling call
0808 20:14:44.151042 (+ 883us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.150740 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:43.438210 15771 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:44.149130 15699 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:44.152732 12244 raft_consensus.cc:2379] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 6: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:44.149204 12243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1932) took 1027 ms. Trace:
I20260808 20:14:44.149432 12142 rpcz_store.cc:276] 0808 20:14:43.039880 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.039996 (+ 116us) service_pool.cc:227] Handling call
0808 20:14:43.512440 (+472444us) spinlock_profiling.cc:243] Waited 1.96 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:44.142710 (+630270us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4512000}
I20260808 20:14:44.153541 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:44.153744 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.154134 15771 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:44.154361 12142 raft_consensus.cc:2454] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:44.154733 12045 raft_consensus.cc:2379] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 6: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:44.148810 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.155335 12047 raft_consensus.cc:2454] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:44.155997 12047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49226 (request call id 514) took 1155 ms. Trace:
I20260808 20:14:44.156296 12047 rpcz_store.cc:276] 0808 20:14:43.000225 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.140527 (+1140302us) service_pool.cc:227] Handling call
0808 20:14:44.155943 (+ 15416us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.152829 15699 raft_consensus.cc:3037] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:44.137413 15776 raft_consensus.cc:504] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:44.157209 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.155251 15726 raft_consensus.cc:2781] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:44.154613 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.142115 12147 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1687) took 1068 ms. Trace:
I20260808 20:14:44.158851 12244 raft_consensus.cc:2379] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 6: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:44.159389 12147 rpcz_store.cc:276] 0808 20:14:43.073855 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.362616 (+288761us) service_pool.cc:227] Handling call
0808 20:14:44.142057 (+779441us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.151829 15667 raft_consensus.cc:504] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:44.160023 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.160292 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.158187 12047 raft_consensus.cc:2396] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260808 20:14:44.158133 15726 raft_consensus.cc:482] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:44.153252 12243 rpcz_store.cc:276] 0808 20:14:43.121930 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.309423 (+187493us) service_pool.cc:227] Handling call
0808 20:14:44.149144 (+839721us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.160754 12244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53030 (request call id 438) took 1656 ms. Trace:
I20260808 20:14:44.148363 12141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 350) took 1094 ms. Trace:
I20260808 20:14:44.161562 12244 rpcz_store.cc:276] 0808 20:14:42.504475 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.159815 (+1655340us) service_pool.cc:227] Handling call
0808 20:14:44.160705 (+ 890us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:43.435387 15832 leader_election.cc:288] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:44.147812 13074 rpcz_store.cc:276] 0808 20:14:41.930413 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.057989 (+1127576us) service_pool.cc:227] Handling call
0808 20:14:44.147345 (+1089356us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.149354 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.162622 15832 raft_consensus.cc:2726] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:44.159353 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 1942) took 1053 ms. Trace:
I20260808 20:14:44.163305 15832 raft_consensus.cc:2781] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:44.163640 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.163702 15832 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:44.164311 15832 raft_consensus.cc:3037] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:44.162797 12039 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:14:44.156565 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.165321 12140 raft_consensus.cc:2454] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:14:44.165686 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.166368 12039 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:44.142350 12251 rpcz_store.cc:276] 0808 20:14:39.257163 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:39.257662 (+ 499us) service_pool.cc:227] Handling call
0808 20:14:39.282779 (+ 25117us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:41.417229 (+2134450us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:41.417256 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:41.442081 (+ 24825us) log.cc:842] Serialized 26 byte log entry
0808 20:14:42.507066 (+1064985us) raft_consensus.cc:1631] Marking committed up to 1
0808 20:14:42.507108 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:42.702479 (+195371us) spinlock_profiling.cc:243] Waited 26.5 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:43.503324 (+800845us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b54000bb3d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:43.503379 (+ 55us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:43.503430 (+ 51us) raft_consensus.cc:1671] finished
0808 20:14:43.503442 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:43.503608 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":296308,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":173,"spinlock_wait_cycles":455733504,"thread_start_us":3008088,"threads_started":2,"wal-append.queue_time_us":612422,"wal-append.run_cpu_time_us":9991,"wal-append.run_wall_time_us":1937992}
I20260808 20:14:44.166787 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 1940) took 1087 ms. Trace:
I20260808 20:14:43.359344 15142 raft_consensus.cc:504] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:44.167204 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:43.359714 15713 consensus_queue.cc:1046] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.808s
I20260808 20:14:44.168299 15726 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:44.167078 12247 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1908) took 4901 ms. Trace:
I20260808 20:14:43.501410 15858 consensus_queue.cc:1046] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.667s
I20260808 20:14:44.154809 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 347) took 1106 ms. Trace:
I20260808 20:14:44.168747 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.169133 12142 rpcz_store.cc:276] 0808 20:14:43.050959 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.153510 (+1102551us) service_pool.cc:227] Handling call
0808 20:14:44.154763 (+ 1253us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.161777 12141 rpcz_store.cc:276] 0808 20:14:43.053491 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.433040 (+379549us) service_pool.cc:227] Handling call
0808 20:14:44.148306 (+715266us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.170059 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:44.170626 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.171243 12141 raft_consensus.cc:2454] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:44.171711 12141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1690) took 1097 ms. Trace:
I20260808 20:14:44.161342 12047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1463) took 1102 ms. Trace:
I20260808 20:14:44.167135 12040 rpcz_store.cc:276] 0808 20:14:43.105637 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.142737 (+1037100us) service_pool.cc:227] Handling call
0808 20:14:44.159300 (+ 16563us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.156729 12045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1447) took 1326 ms. Trace:
I20260808 20:14:44.172732 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.168236 15646 consensus_queue.cc:260] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:44.172556 12045 rpcz_store.cc:276] 0808 20:14:42.830631 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:42.910725 (+ 80094us) service_pool.cc:227] Handling call
0808 20:14:44.156562 (+1245837us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54004853d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:44.156677 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2645639936}
I20260808 20:14:44.158782 15776 leader_election.cc:288] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:44.161242 15667 leader_election.cc:288] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:44.167085 12039 rpcz_store.cc:276] 0808 20:14:43.079314 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.165456 (+1086142us) service_pool.cc:227] Handling call
0808 20:14:44.166742 (+ 1286us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.169687 12245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1757) took 1155 ms. Trace:
I20260808 20:14:44.168938 12247 rpcz_store.cc:276] 0808 20:14:39.265540 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:39.265676 (+ 136us) service_pool.cc:227] Handling call
0808 20:14:39.266017 (+ 341us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 20:14:40.588461 (+1322444us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:40.588488 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 20:14:40.653987 (+ 65499us) log.cc:842] Serialized 52 byte log entry
0808 20:14:42.714080 (+2060093us) raft_consensus.cc:1631] Marking committed up to 1
0808 20:14:42.714125 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:42.714185 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:43.064349 (+350164us) spinlock_profiling.cc:243] Waited 3.47 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:44.166766 (+1102417us) raft_consensus.cc:1671] finished
0808 20:14:44.166789 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:44.167021 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":368386,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":8965,"spinlock_wait_cycles":7970048,"thread_start_us":2690885,"threads_started":2,"wal-append.queue_time_us":1715607}
I20260808 20:14:44.174846 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.176292 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.166117 12101 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:44.173718 12040 raft_consensus.cc:2454] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:44.177389 12045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49226 (request call id 515) took 1155 ms. Trace:
I20260808 20:14:44.177398 12247 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:14:44.178529 12247 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1926) took 1152 ms. Trace:
I20260808 20:14:44.175721 12245 rpcz_store.cc:276] 0808 20:14:43.014525 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.139402 (+1124877us) service_pool.cc:227] Handling call
0808 20:14:44.169637 (+ 30235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.173182 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.178454 12047 rpcz_store.cc:276] 0808 20:14:43.058943 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.156952 (+1098009us) service_pool.cc:227] Handling call
0808 20:14:44.161293 (+ 4341us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.163091 12254 raft_consensus.cc:2379] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 6: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:44.178992 12247 rpcz_store.cc:276] 0808 20:14:43.026860 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.176025 (+1149165us) service_pool.cc:227] Handling call
0808 20:14:44.178479 (+ 2454us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.179981 12152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1395) took 1139 ms. Trace:
I20260808 20:14:44.179680 12048 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6.
I20260808 20:14:44.149464 15873 raft_consensus.cc:504] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:44.178342 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.165728 12140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 354) took 1111 ms. Trace:
I20260808 20:14:44.176734 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:44.177717 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 811558us
I20260808 20:14:44.180581 12048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49226 (request call id 517) took 1102 ms. Trace:
I20260808 20:14:44.179358 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.180459 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.180367 12152 rpcz_store.cc:276] 0808 20:14:43.040556 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.040665 (+ 109us) service_pool.cc:227] Handling call
0808 20:14:44.179927 (+1139262us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.177523 12147 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 349) took 1124 ms. Trace:
I20260808 20:14:44.168133 12251 raft_consensus.cc:2379] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:44.175742 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.178141 12045 rpcz_store.cc:276] 0808 20:14:43.021901 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.174607 (+1152706us) service_pool.cc:227] Handling call
0808 20:14:44.177340 (+ 2733us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.170702 12142 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Advancing to term 9
W20260808 20:14:43.518811 15875 log.cc:924] Time spent T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.537s user 0.003s sys 0.003s
I20260808 20:14:44.174279 15646 raft_consensus.cc:3037] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:44.183954 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 503) took 3502 ms. Trace:
I20260808 20:14:43.390816 12138 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:44.239712 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.242035 15667 raft_consensus.cc:2726] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:44.235116 12036 rpcz_store.cc:276] 0808 20:14:40.681646 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:42.032529 (+1350883us) service_pool.cc:227] Handling call
0808 20:14:42.045950 (+ 13421us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:43.535661 (+1489711us) spinlock_profiling.cc:243] Waited 530 ms on lock 0x7b5400241e20. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa968270951 00007fa9682d1321 00007fa9682d2b4e 00007fa9682dcc89 00007fa9682dd556 00007fa9682d9edf 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612
0808 20:14:44.183439 (+647778us) raft_consensus.cc:1520] Early marking committed up to index 4
0808 20:14:44.183472 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:44.183514 (+ 42us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:44.183542 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.183755 (+ 213us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:44.183898 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":92,"spinlock_wait_cycles":1218331520}
I20260808 20:14:44.242412 12244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1762) took 1226 ms. Trace:
I20260808 20:14:43.422294 15690 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:44.105031 15870 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:44.242760 12244 rpcz_store.cc:276] 0808 20:14:43.018517 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.179182 (+1160665us) service_pool.cc:227] Handling call
0808 20:14:44.242352 (+ 63170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:43.517081 15614 raft_consensus.cc:3032] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 20:14:44.243865 15614 raft_consensus.cc:729] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
W20260808 20:14:44.086405 15624 consensus_meta.cc:287] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.956s user 0.007s sys 0.002s
I20260808 20:14:44.244107 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.236128 12251 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1764) took 1181 ms. Trace:
I20260808 20:14:44.245146 12244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1923) took 1246 ms. Trace:
I20260808 20:14:44.234926 12039 raft_consensus.cc:2396] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 20:14:44.245591 12244 rpcz_store.cc:276] 0808 20:14:42.998971 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.243875 (+1244904us) service_pool.cc:227] Handling call
0808 20:14:44.245095 (+ 1220us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.236547 12137 raft_consensus.cc:2454] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8.
I20260808 20:14:44.245100 12251 rpcz_store.cc:276] 0808 20:14:43.054329 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.166971 (+1112642us) service_pool.cc:227] Handling call
0808 20:14:44.236073 (+ 69102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.235823 12245 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6.
I20260808 20:14:44.237056 12048 rpcz_store.cc:276] 0808 20:14:43.078412 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.130368 (+1051956us) service_pool.cc:227] Handling call
0808 20:14:44.180533 (+ 50165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.236779 12147 rpcz_store.cc:276] 0808 20:14:43.053206 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.160060 (+1106854us) service_pool.cc:227] Handling call
0808 20:14:44.177468 (+ 17408us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.238147 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:44.183281 15886 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.651s user 0.001s sys 0.009s
I20260808 20:14:44.172057 12141 rpcz_store.cc:276] 0808 20:14:43.074662 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.170377 (+1095715us) service_pool.cc:227] Handling call
0808 20:14:44.171661 (+ 1284us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.240485 15835 raft_consensus.cc:482] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:44.239352 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:44.262740 15835 raft_consensus.cc:504] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:44.263120 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.164381 12243 raft_consensus.cc:2454] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:44.265982 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.266628 12045 raft_consensus.cc:2454] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:44.174489 15142 leader_election.cc:288] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:44.267578 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.171869 15832 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:44.238298 12140 rpcz_store.cc:276] 0808 20:14:43.054715 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.156339 (+1101624us) service_pool.cc:227] Handling call
0808 20:14:44.165683 (+ 9344us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.244019 12041 raft_consensus.cc:2379] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 5: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:44.247963 12137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1691) took 1173 ms. Trace:
I20260808 20:14:44.250530 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.231530 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1391) took 1101 ms. Trace:
I20260808 20:14:44.269665 12137 rpcz_store.cc:276] 0808 20:14:43.075349 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.176468 (+1101119us) service_pool.cc:227] Handling call
0808 20:14:44.247895 (+ 71427us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.256261 12248 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:44.256800 12250 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1907) took 4993 ms. Trace:
I20260808 20:14:44.257028 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.273550 12250 rpcz_store.cc:276] 0808 20:14:39.263093 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:39.263249 (+ 156us) service_pool.cc:227] Handling call
0808 20:14:39.263587 (+ 338us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:41.932245 (+2668658us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:41.932276 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:41.952237 (+ 19961us) log.cc:842] Serialized 26 byte log entry
0808 20:14:42.297168 (+344931us) raft_consensus.cc:1631] Marking committed up to 1
0808 20:14:42.297202 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:42.297332 (+ 130us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:43.518184 (+1220852us) spinlock_profiling.cc:243] Waited 2.07 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:44.246720 (+728536us) raft_consensus.cc:1671] finished
0808 20:14:44.246744 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:44.256737 (+ 9993us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":357832,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":8114,"spinlock_wait_cycles":4764928,"thread_start_us":2928319,"threads_started":2,"wal-append.queue_time_us":852859}
I20260808 20:14:44.275115 12244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1927) took 1243 ms. Trace:
I20260808 20:14:44.275629 12250 raft_consensus.cc:1226] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:44.257597 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.275434 12244 rpcz_store.cc:276] 0808 20:14:43.031567 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.246255 (+1214688us) service_pool.cc:227] Handling call
0808 20:14:44.275057 (+ 28802us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.276773 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.277457 12045 raft_consensus.cc:2454] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 1.
I20260808 20:14:44.277616 12043 raft_consensus.cc:2379] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 6: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:44.258358 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.258755 12254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1919) took 1275 ms. Trace:
I20260808 20:14:44.278234 12254 rpcz_store.cc:276] 0808 20:14:42.983762 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.180255 (+1196493us) service_pool.cc:227] Handling call
0808 20:14:44.258688 (+ 78433us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.236574 12040 raft_consensus.cc:2454] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:44.279194 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49226 (request call id 507) took 1299 ms. Trace:
I20260808 20:14:44.279526 12040 rpcz_store.cc:276] 0808 20:14:42.979388 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.178134 (+1198746us) service_pool.cc:227] Handling call
0808 20:14:44.279134 (+101000us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.280057 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.257993 15624 raft_consensus.cc:482] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:44.261504 12245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 429) took 2194 ms. Trace:
I20260808 20:14:44.263334 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.266490 12251 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:14:44.285375 12047 raft_consensus.cc:2396] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260808 20:14:43.533958 15853 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:44.182085 15675 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:44.135954 15650 raft_consensus.cc:2781] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260808 20:14:43.501113 15845 log.cc:924] Time spent T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 1.084s user 0.001s sys 0.007s
I20260808 20:14:44.264673 12247 raft_consensus.cc:2396] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 20:14:44.270181 12146 rpcz_store.cc:276] 0808 20:14:43.039616 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.039736 (+ 120us) service_pool.cc:227] Handling call
0808 20:14:43.504393 (+464657us) spinlock_profiling.cc:243] Waited 1.32 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:43.504619 (+ 226us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:44.141118 (+636499us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.141308 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3027200}
I20260808 20:14:44.271088 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.300166 12147 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 344) took 1252 ms. Trace:
I20260808 20:14:44.251456 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 1934) took 1221 ms. Trace:
I20260808 20:14:44.300935 12039 rpcz_store.cc:276] 0808 20:14:43.030433 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.175471 (+1145038us) service_pool.cc:227] Handling call
0808 20:14:44.251392 (+ 75921us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:44.259389 15886 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.762s user 0.001s sys 0.009s
I20260808 20:14:44.280375 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.237525 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:44.281148 12043 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 1943) took 1092 ms. Trace:
I20260808 20:14:44.281782 12244 raft_consensus.cc:1261] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 20:14:44.282366 12250 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 430) took 2207 ms. Trace:
I20260808 20:14:44.283372 12254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1911) took 1399 ms. Trace:
I20260808 20:14:44.283910 12048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 1928) took 1258 ms. Trace:
I20260808 20:14:44.284775 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.283861 12038 raft_consensus.cc:2454] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6.
I20260808 20:14:44.284557 12245 rpcz_store.cc:276] 0808 20:14:42.066889 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.247308 (+2180419us) service_pool.cc:227] Handling call
0808 20:14:44.260835 (+ 13527us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.261193 (+ 358us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.261221 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:44.261259 (+ 38us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:44.261285 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.261332 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:44.261429 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1672,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":94}
I20260808 20:14:44.303678 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49226 (request call id 513) took 1305 ms. Trace:
I20260808 20:14:44.304013 12038 rpcz_store.cc:276] 0808 20:14:42.998736 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.258045 (+1259309us) service_pool.cc:227] Handling call
0808 20:14:44.303619 (+ 45574us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.304365 12048 rpcz_store.cc:276] 0808 20:14:43.025488 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.268256 (+1242768us) service_pool.cc:227] Handling call
0808 20:14:44.283844 (+ 15588us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.305356 12152 raft_consensus.cc:3037] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:44.303117 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.306596 12036 raft_consensus.cc:2396] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260808 20:14:44.303102 12250 rpcz_store.cc:276] 0808 20:14:42.091915 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.275184 (+2183269us) service_pool.cc:227] Handling call
0808 20:14:44.275466 (+ 282us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.282106 (+ 6640us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.282310 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.308715 12250 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1916) took 1382 ms. Trace:
W20260808 20:14:44.309159 13016 leader_election.cc:334] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.738s (SENT)
I20260808 20:14:44.302386 12043 rpcz_store.cc:276] 0808 20:14:43.216145 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.179784 (+963639us) service_pool.cc:227] Handling call
0808 20:14:44.281075 (+101291us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.309104 12250 rpcz_store.cc:276] 0808 20:14:42.926127 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.308013 (+1381886us) service_pool.cc:227] Handling call
0808 20:14:44.308281 (+ 268us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.308451 (+ 170us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.308474 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:44.308511 (+ 37us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:44.308538 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.308581 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:44.308666 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.309777 13016 leader_election.cc:302] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:44.311041 12250 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1917) took 1327 ms. Trace:
I20260808 20:14:44.311678 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1920) took 1688 ms. Trace:
I20260808 20:14:44.285506 12251 raft_consensus.cc:595] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:44.303328 12254 rpcz_store.cc:276] 0808 20:14:42.884073 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.278855 (+1394782us) service_pool.cc:227] Handling call
0808 20:14:44.282913 (+ 4058us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.283090 (+ 177us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.283114 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:44.283161 (+ 47us) raft_consensus.cc:1631] Marking committed up to 1
0808 20:14:44.283190 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.283235 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:44.283323 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.312271 12038 rpcz_store.cc:276] 0808 20:14:42.623067 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.310909 (+1687842us) service_pool.cc:227] Handling call
0808 20:14:44.311208 (+ 299us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.311371 (+ 163us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.311395 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:44.311456 (+ 61us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:44.311484 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.311533 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:44.311623 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.270772 15666 raft_consensus.cc:2781] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:44.315577 12247 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1924) took 1315 ms. Trace:
I20260808 20:14:44.315979 12247 rpcz_store.cc:276] 0808 20:14:43.000264 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.181180 (+1180916us) service_pool.cc:227] Handling call
0808 20:14:44.315518 (+134338us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.300064 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.290295 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.318084 12137 raft_consensus.cc:2454] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:44.318554 12137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1682) took 1256 ms. Trace:
I20260808 20:14:44.300557 12147 rpcz_store.cc:276] 0808 20:14:43.047797 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.257604 (+1209807us) service_pool.cc:227] Handling call
0808 20:14:44.300098 (+ 42494us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.318825 12137 rpcz_store.cc:276] 0808 20:14:43.061681 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.273100 (+1211419us) service_pool.cc:227] Handling call
0808 20:14:44.318497 (+ 45397us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.316869 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49226 (request call id 508) took 1332 ms. Trace:
I20260808 20:14:44.319715 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.320139 12040 rpcz_store.cc:276] 0808 20:14:42.984758 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.280138 (+1295380us) service_pool.cc:227] Handling call
0808 20:14:44.316809 (+ 36671us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.320547 12137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1400) took 1278 ms. Trace:
I20260808 20:14:44.321029 12137 rpcz_store.cc:276] 0808 20:14:43.041945 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.319467 (+1277522us) service_pool.cc:227] Handling call
0808 20:14:44.320492 (+ 1025us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.314471 12038 raft_consensus.cc:1261] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 20:14:44.314965 12140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1401) took 1260 ms. Trace:
I20260808 20:14:44.289204 12047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 1936) took 1249 ms. Trace:
I20260808 20:14:44.322474 12140 rpcz_store.cc:276] 0808 20:14:43.054898 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.272086 (+1217188us) service_pool.cc:227] Handling call
0808 20:14:44.314894 (+ 42808us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.322576 12047 rpcz_store.cc:276] 0808 20:14:43.039910 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.263105 (+1223195us) service_pool.cc:227] Handling call
0808 20:14:44.289143 (+ 26038us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.322458 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.321766 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1923) took 1697 ms. Trace:
I20260808 20:14:44.325086 12040 rpcz_store.cc:276] 0808 20:14:42.623957 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.321073 (+1697116us) service_pool.cc:227] Handling call
0808 20:14:44.321347 (+ 274us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.321502 (+ 155us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.321524 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:44.321562 (+ 38us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:44.321589 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.321633 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:44.321718 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.311393 12250 rpcz_store.cc:276] 0808 20:14:42.983176 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.310365 (+1327189us) service_pool.cc:227] Handling call
0808 20:14:44.310608 (+ 243us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.310754 (+ 146us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.310780 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:44.310819 (+ 39us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:44.310850 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.310896 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:44.310993 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.327638 12250 raft_consensus.cc:1226] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 6. Current term is 7. Ops: []
I20260808 20:14:44.328265 12250 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1758) took 1313 ms. Trace:
I20260808 20:14:44.328624 12250 rpcz_store.cc:276] 0808 20:14:43.014821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.327193 (+1312372us) service_pool.cc:227] Handling call
0808 20:14:44.327469 (+ 276us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.328063 (+ 594us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.328217 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.329851 12250 raft_consensus.cc:1226] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 7. Current term is 8. Ops: []
I20260808 20:14:44.330443 12250 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1761) took 1314 ms. Trace:
I20260808 20:14:44.324045 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1922) took 1700 ms. Trace:
I20260808 20:14:44.331009 12038 rpcz_store.cc:276] 0808 20:14:42.623704 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.313846 (+1690142us) service_pool.cc:227] Handling call
0808 20:14:44.314140 (+ 294us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.323823 (+ 9683us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.324000 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.324617 12137 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:44.332600 12137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1693) took 1256 ms. Trace:
I20260808 20:14:44.333164 12137 rpcz_store.cc:276] 0808 20:14:43.075935 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.322225 (+1246290us) service_pool.cc:227] Handling call
0808 20:14:44.332545 (+ 10320us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.334086 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:44.334689 12137 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:44.336198 12250 rpcz_store.cc:276] 0808 20:14:43.015632 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.329420 (+1313788us) service_pool.cc:227] Handling call
0808 20:14:44.329696 (+ 276us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.330246 (+ 550us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.330398 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:44.306191 15674 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.809s user 0.013s sys 0.058s
I20260808 20:14:44.306324 12244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 437) took 1802 ms. Trace:
I20260808 20:14:44.349933 12152 raft_consensus.cc:2454] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:14:44.351683 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.282588 13076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 484) took 1105 ms. Trace:
I20260808 20:14:44.344815 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1926) took 1716 ms. Trace:
I20260808 20:14:44.129240 13072 rpcz_store.cc:276] 0808 20:14:41.434994 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.157670 (+1722676us) service_pool.cc:227] Handling call
0808 20:14:43.319962 (+162292us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:43.320173 (+ 211us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:43.320201 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:43.320252 (+ 51us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:43.320286 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:43.320342 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:43.320454 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.355607 12038 rpcz_store.cc:276] 0808 20:14:42.624727 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.340002 (+1715275us) service_pool.cc:227] Handling call
0808 20:14:44.340278 (+ 276us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.340450 (+ 172us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.340474 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:44.340515 (+ 41us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:44.340543 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.340589 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:44.340674 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.356668 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.357193 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1448) took 1526 ms. Trace:
I20260808 20:14:44.357419 12140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1686) took 1283 ms. Trace:
I20260808 20:14:44.352159 12152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1405) took 1290 ms. Trace:
I20260808 20:14:44.358016 12152 rpcz_store.cc:276] 0808 20:14:43.061669 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.237224 (+1175555us) service_pool.cc:227] Handling call
0808 20:14:44.352103 (+114879us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.358829 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:44.359373 12040 raft_consensus.cc:1261] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20260808 20:14:44.359454 12152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 345) took 1311 ms. Trace:
I20260808 20:14:44.352298 12146 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:44.360083 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1685) took 1286 ms. Trace:
I20260808 20:14:44.342332 12245 raft_consensus.cc:1261] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260808 20:14:44.352972 12244 rpcz_store.cc:276] 0808 20:14:42.504209 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.276961 (+1772752us) service_pool.cc:227] Handling call
0808 20:14:44.281477 (+ 4516us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.302797 (+ 21320us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.306269 (+ 3472us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.360261 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.360963 12048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 1933) took 1330 ms. Trace:
I20260808 20:14:44.361258 12048 rpcz_store.cc:276] 0808 20:14:43.030147 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.305714 (+1275567us) service_pool.cc:227] Handling call
0808 20:14:44.360886 (+ 55172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.357733 12140 rpcz_store.cc:276] 0808 20:14:43.073525 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.323468 (+1249943us) service_pool.cc:227] Handling call
0808 20:14:44.357368 (+ 33900us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.352929 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.362600 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
W20260808 20:14:44.355001 15674 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.956s user 0.013s sys 0.058s
I20260808 20:14:44.363473 12146 rpcz_store.cc:276] 0808 20:14:43.073221 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.299683 (+1226462us) service_pool.cc:227] Handling call
0808 20:14:44.360033 (+ 60350us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.364364 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.364988 12146 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:44.365379 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1407) took 1260 ms. Trace:
I20260808 20:14:44.365651 12146 rpcz_store.cc:276] 0808 20:14:43.105222 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.364138 (+1258916us) service_pool.cc:227] Handling call
0808 20:14:44.365334 (+ 1196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.366402 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.369310 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:44.369879 12147 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:44.362603 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 1930) took 1336 ms. Trace:
I20260808 20:14:44.373106 12039 rpcz_store.cc:276] 0808 20:14:43.025845 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.301557 (+1275712us) service_pool.cc:227] Handling call
0808 20:14:44.362549 (+ 60992us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.373850 12245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1915) took 1458 ms. Trace:
I20260808 20:14:44.374130 12247 raft_consensus.cc:1261] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 20:14:44.374776 12247 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1920) took 1384 ms. Trace:
I20260808 20:14:44.375293 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1452) took 1531 ms. Trace:
I20260808 20:14:44.375638 12039 rpcz_store.cc:276] 0808 20:14:42.844076 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.374611 (+1530535us) service_pool.cc:227] Handling call
0808 20:14:44.374885 (+ 274us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.375029 (+ 144us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.375052 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:44.375088 (+ 36us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:44.375118 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.375165 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:44.375247 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.375056 12247 rpcz_store.cc:276] 0808 20:14:42.990113 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.316579 (+1326466us) service_pool.cc:227] Handling call
0808 20:14:44.373826 (+ 57247us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.374572 (+ 746us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.374731 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.377466 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.378000 12247 raft_consensus.cc:2454] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:14:44.378873 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.379496 12247 raft_consensus.cc:2379] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:44.380158 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.385874 15835 leader_election.cc:288] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:44.388744 15699 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:44.389518 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.390069 12251 raft_consensus.cc:2454] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:44.390846 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.391351 12251 raft_consensus.cc:2454] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:44.392086 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.292835 15650 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:44.373487 12047 raft_consensus.cc:1261] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 20:14:44.395558 12152 rpcz_store.cc:276] 0808 20:14:43.048051 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.358608 (+1310557us) service_pool.cc:227] Handling call
0808 20:14:44.359407 (+ 799us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.426994 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1689) took 1352 ms. Trace:
I20260808 20:14:44.374162 12245 rpcz_store.cc:276] 0808 20:14:42.915577 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.303935 (+1388358us) service_pool.cc:227] Handling call
0808 20:14:44.341989 (+ 38054us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.373610 (+ 31621us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.373799 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.359742 12040 pending_rounds.cc:76] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting all ops after (but not including) 5
W20260808 20:14:44.354745 15743 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.872s user 0.043s sys 0.067s
I20260808 20:14:44.380788 12247 raft_consensus.cc:2379] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 8: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:44.405441 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:44.300770 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.392809 12251 raft_consensus.cc:2396] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 20:14:44.428004 12146 rpcz_store.cc:276] 0808 20:14:43.074402 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.366177 (+1291775us) service_pool.cc:227] Handling call
0808 20:14:44.426939 (+ 60762us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.431218 12040 pending_rounds.cc:97] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting uncommitted NO_OP operation due to leader change: 7.6
I20260808 20:14:44.427983 12038 rpcz_store.cc:276] 0808 20:14:42.830864 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.356489 (+1525625us) service_pool.cc:227] Handling call
0808 20:14:44.356741 (+ 252us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.356917 (+ 176us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.356940 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:44.356981 (+ 41us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:44.357008 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.357057 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:44.357148 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.412492 12140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 357) took 1355 ms. Trace:
I20260808 20:14:44.427861 12047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1924) took 1803 ms. Trace:
I20260808 20:14:44.409329 12137 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8.
I20260808 20:14:44.438102 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.503319 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1455) took 1589 ms. Trace:
I20260808 20:14:44.501293 12140 rpcz_store.cc:276] 0808 20:14:43.056921 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.362392 (+1305471us) service_pool.cc:227] Handling call
0808 20:14:44.412416 (+ 50024us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.504601 12150 raft_consensus.cc:1226] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 7. Current term is 8. Ops: [5.5-7.6]
I20260808 20:14:44.505033 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.505290 12137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1688) took 1431 ms. Trace:
I20260808 20:14:44.505786 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1694) took 1429 ms. Trace:
I20260808 20:14:44.506147 12146 rpcz_store.cc:276] 0808 20:14:43.076246 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.501625 (+1425379us) service_pool.cc:227] Handling call
0808 20:14:44.505732 (+ 4107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.506965 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.507529 12146 raft_consensus.cc:2454] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:44.507951 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1397) took 1466 ms. Trace:
I20260808 20:14:44.508219 12146 rpcz_store.cc:276] 0808 20:14:43.041110 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.506716 (+1465606us) service_pool.cc:227] Handling call
0808 20:14:44.507908 (+ 1192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.505664 12137 rpcz_store.cc:276] 0808 20:14:43.074144 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.333846 (+1259702us) service_pool.cc:227] Handling call
0808 20:14:44.505230 (+171384us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.509232 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.511638 12142 raft_consensus.cc:2454] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 9.
I20260808 20:14:44.511708 12048 raft_consensus.cc:1226] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: []
W20260808 20:14:44.512220 12142 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1408) took 1380 ms (client timeout 1728 ms). Trace:
I20260808 20:14:44.512471 12048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1449) took 1681 ms. Trace:
I20260808 20:14:44.503697 12036 raft_consensus.cc:2379] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 8: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
W20260808 20:14:44.512584 12142 rpcz_store.cc:269] 0808 20:14:43.131546 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.169807 (+1038261us) service_pool.cc:227] Handling call
0808 20:14:44.512165 (+342358us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.512871 12048 rpcz_store.cc:276] 0808 20:14:42.831108 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.361798 (+1530690us) service_pool.cc:227] Handling call
0808 20:14:44.511588 (+149790us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.512300 (+ 712us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.512433 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.501122 12047 rpcz_store.cc:276] 0808 20:14:42.624251 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.323744 (+1699493us) service_pool.cc:227] Handling call
0808 20:14:44.373192 (+ 49448us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.427622 (+ 54430us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.427810 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.433310 12045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1927) took 1808 ms. Trace:
I20260808 20:14:44.514988 12045 rpcz_store.cc:276] 0808 20:14:42.624991 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.345357 (+1720366us) service_pool.cc:227] Handling call
0808 20:14:44.432842 (+ 87485us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.433029 (+ 187us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.433053 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:44.433093 (+ 40us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:44.433117 (+ 24us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.433163 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:44.433255 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.515950 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 1937) took 1465 ms. Trace:
I20260808 20:14:44.517577 12152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1692) took 1442 ms. Trace:
I20260808 20:14:44.517426 12036 rpcz_store.cc:276] 0808 20:14:43.051042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.307664 (+1256622us) service_pool.cc:227] Handling call
0808 20:14:44.515889 (+208225us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.518023 12152 rpcz_store.cc:276] 0808 20:14:43.075655 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.437856 (+1362201us) service_pool.cc:227] Handling call
0808 20:14:44.517515 (+ 79659us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.294548 15624 raft_consensus.cc:504] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:44.503700 12038 rpcz_store.cc:276] 0808 20:14:42.914413 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.502510 (+1588097us) service_pool.cc:227] Handling call
0808 20:14:44.502853 (+ 343us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.503026 (+ 173us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.503051 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:44.503089 (+ 38us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:44.503118 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.503164 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:44.503267 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.510433 12039 pending_rounds.cc:76] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting all ops after (but not including) 5
I20260808 20:14:44.522192 12039 pending_rounds.cc:97] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260808 20:14:44.522451 12039 raft_consensus.cc:2866] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:44.519843 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:44.523864 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1396) took 1483 ms. Trace:
I20260808 20:14:44.524242 12146 rpcz_store.cc:276] 0808 20:14:43.040814 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.508724 (+1467910us) service_pool.cc:227] Handling call
0808 20:14:44.523812 (+ 15088us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.501292 12040 raft_consensus.cc:2866] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:44.525550 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1925) took 1901 ms. Trace:
I20260808 20:14:44.525943 12040 rpcz_store.cc:276] 0808 20:14:42.624496 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.325998 (+1701502us) service_pool.cc:227] Handling call
0808 20:14:44.326281 (+ 283us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.525323 (+199042us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.525498 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:44.435843 12248 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 1929) took 1378 ms (client timeout 1824 ms). Trace:
W20260808 20:14:44.526993 12248 rpcz_store.cc:269] 0808 20:14:43.056977 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.158826 (+101849us) service_pool.cc:227] Handling call
0808 20:14:44.435685 (+1276859us) spinlock_profiling.cc:243] Waited 907 ms on lock 0x7b540029ed54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:44.435781 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2086642944}
W20260808 20:14:44.434253 15743 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.972s user 0.043s sys 0.067s
I20260808 20:14:44.529269 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:44.362017 15873 leader_election.cc:288] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:44.531551 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.549907 12046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 505) took 3725 ms. Trace:
I20260808 20:14:44.532583 12040 raft_consensus.cc:1226] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 7. Current term is 8. Ops: []
I20260808 20:14:44.433845 15650 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:44.550725 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:44.533804 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 409713us
I20260808 20:14:44.533818 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.534615 12047 raft_consensus.cc:1226] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:44.553676 12047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1929) took 1528 ms. Trace:
I20260808 20:14:44.339613 15726 raft_consensus.cc:504] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:44.536484 12045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1461) took 1505 ms. Trace:
I20260808 20:14:44.317399 15614 consensus_queue.cc:260] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:44.543799 12147 raft_consensus.cc:2454] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:14:44.544450 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.554091 12040 pending_rounds.cc:76] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting all ops after (but not including) 5
I20260808 20:14:44.677028 12040 pending_rounds.cc:97] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260808 20:14:44.517164 15832 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:44.675781 12045 rpcz_store.cc:276] 0808 20:14:43.030113 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.518822 (+1488709us) service_pool.cc:227] Handling call
0808 20:14:44.535546 (+ 16724us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.535721 (+ 175us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.535743 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:44.535780 (+ 37us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:14:44.535808 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.535854 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:44.535940 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.677511 12040 raft_consensus.cc:2866] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 20:14:44.678074 12147 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1406) took 1614 ms (client timeout 1997 ms). Trace:
I20260808 20:14:44.550264 12046 rpcz_store.cc:276] 0808 20:14:40.824938 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:42.505154 (+1680216us) service_pool.cc:227] Handling call
0808 20:14:42.534429 (+ 29275us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:42.610900 (+ 76471us) raft_consensus.cc:1520] Early marking committed up to index 3
0808 20:14:42.610933 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:42.610973 (+ 40us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:42.611012 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.549707 (+1938695us) spinlock_profiling.cc:243] Waited 76.2 ms on lock 0x7b54004646d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:44.549750 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:44.549857 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":175323904}
I20260808 20:14:44.556427 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:44.677752 13015 leader_election.cc:334] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Timed out: RequestConsensusVote RPC to 127.10.24.1:40747 timed out after 1.793s (SENT)
W20260808 20:14:44.681099 12147 rpcz_store.cc:269] 0808 20:14:43.063696 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.319333 (+1255637us) service_pool.cc:227] Handling call
0808 20:14:44.678011 (+358678us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.682260 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.683013 12147 raft_consensus.cc:2379] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:44.677342 12146 raft_consensus.cc:2396] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 20:14:44.553998 12047 rpcz_store.cc:276] 0808 20:14:43.025686 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.516682 (+1490996us) service_pool.cc:227] Handling call
0808 20:14:44.534459 (+ 17777us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.553468 (+ 19009us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.553630 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.683759 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 356) took 1628 ms. Trace:
I20260808 20:14:44.683350 12150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1388) took 1650 ms. Trace:
I20260808 20:14:44.683347 12147 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 351) took 1629 ms. Trace:
I20260808 20:14:44.681846 13015 leader_election.cc:302] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:44.684746 12147 rpcz_store.cc:276] 0808 20:14:43.053888 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.681990 (+1628102us) service_pool.cc:227] Handling call
0808 20:14:44.683297 (+ 1307us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.684091 12146 rpcz_store.cc:276] 0808 20:14:43.055206 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.524918 (+1469712us) service_pool.cc:227] Handling call
0808 20:14:44.683705 (+158787us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:44.684085 12041 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 1939) took 1606 ms (client timeout 1824 ms). Trace:
W20260808 20:14:44.686347 12041 rpcz_store.cc:269] 0808 20:14:43.077442 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.083355 (+ 5913us) service_pool.cc:227] Handling call
0808 20:14:44.683909 (+1600554us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b5400465ad4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:44.684032 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2657602432}
I20260808 20:14:44.685534 12150 rpcz_store.cc:276] 0808 20:14:43.033001 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.033137 (+ 136us) service_pool.cc:227] Handling call
0808 20:14:44.435117 (+1401980us) spinlock_profiling.cc:243] Waited 1.89 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:44.504358 (+ 69241us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 20:14:44.505009 (+ 651us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.683180 (+178171us) spinlock_profiling.cc:243] Waited 69 ms on lock 0x7b54004254d0. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d9a09 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:44.683295 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":163093120}
W20260808 20:14:44.686913 13015 leader_election.cc:334] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Timed out: RequestConsensusVote RPC to 127.10.24.1:40747 timed out after 2.113s (SENT)
I20260808 20:14:44.384881 15891 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:44.689539 13015 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:44.554934 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 415797us
W20260808 20:14:44.693837 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 379760us
W20260808 20:14:44.697413 12249 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53030 (request call id 426) took 2657 ms (client timeout 1738 ms). Trace:
W20260808 20:14:44.697819 12249 rpcz_store.cc:269] 0808 20:14:42.030688 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:42.030914 (+ 226us) service_pool.cc:227] Handling call
0808 20:14:44.687398 (+2656484us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b5400374954. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:44.687503 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3765545216}
W20260808 20:14:44.698719 13016 leader_election.cc:334] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.839s (SENT)
W20260808 20:14:44.691205 12105 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.006s sys 0.017s
I20260808 20:14:44.701527 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:44.702397 12045 pending_rounds.cc:76] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting all ops after (but not including) 5
I20260808 20:14:44.702730 12045 pending_rounds.cc:97] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260808 20:14:44.702963 12045 raft_consensus.cc:2866] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 20:14:44.700634 12105 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.555s user 0.006s sys 0.017s
W20260808 20:14:44.704561 13014 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.004s sys 0.022s
W20260808 20:14:44.704753 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 559777us
W20260808 20:14:44.705201 13014 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.543s user 0.004s sys 0.022s
W20260808 20:14:44.686048 15653 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.004s sys 0.013s
I20260808 20:14:44.721328 12150 raft_consensus.cc:1261] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 20:14:44.721952 12150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 340) took 1675 ms. Trace:
I20260808 20:14:44.676103 12243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1912) took 1659 ms. Trace:
I20260808 20:14:44.559707 15690 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:44.718451 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.355536 13076 rpcz_store.cc:276] 0808 20:14:43.173744 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.462143 (+288399us) service_pool.cc:227] Handling call
0808 20:14:44.132459 (+670316us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.250296 (+117837us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.251580 (+ 1284us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:44.259819 (+ 8239us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:44.264119 (+ 4300us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.268619 (+ 4500us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:44.279096 (+ 10477us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.717398 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.747830 12147 raft_consensus.cc:2454] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:44.748311 12147 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 346) took 1700 ms. Trace:
W20260808 20:14:44.723114 15653 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.552s user 0.005s sys 0.013s
I20260808 20:14:44.722312 12150 rpcz_store.cc:276] 0808 20:14:43.046802 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.687636 (+1640834us) service_pool.cc:227] Handling call
0808 20:14:44.716691 (+ 29055us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.721713 (+ 5022us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.721898 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.718007 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.552292 15592 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:44.702076 12247 raft_consensus.cc:2362] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5. Re-sending same reply.
I20260808 20:14:44.716864 15614 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:44.719844 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:44.754557 12248 raft_consensus.cc:2396] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20260808 20:14:44.729722 12102 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.011s sys 0.016s
W20260808 20:14:44.754906 12102 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.615s user 0.011s sys 0.016s
W20260808 20:14:44.755450 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 616467us
I20260808 20:14:44.744380 12245 raft_consensus.cc:2454] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:44.756316 12102 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:44.694550 15699 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:44.552707 12141 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 355) took 1494 ms (client timeout 1839 ms). Trace:
I20260808 20:14:44.746043 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:44.748594 12147 rpcz_store.cc:276] 0808 20:14:43.048272 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.685467 (+1637195us) service_pool.cc:227] Handling call
0808 20:14:44.748254 (+ 62787us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:44.705421 15852 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.010s sys 0.014s
I20260808 20:14:44.751978 12105 leader_election.cc:302] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:44.761144 12102 leader_election.cc:302] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:44.793781 12102 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:44.795495 12102 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:44.752044 15713 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.007s sys 0.021s
I20260808 20:14:44.802421 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.803058 12245 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8.
I20260808 20:14:44.800227 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:44.796792 12102 leader_election.cc:302] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:44.799387 12150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 341) took 1752 ms. Trace:
I20260808 20:14:44.807209 12150 rpcz_store.cc:276] 0808 20:14:43.047092 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.758449 (+1711357us) service_pool.cc:227] Handling call
0808 20:14:44.798869 (+ 40420us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.799058 (+ 189us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.799082 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:44.799131 (+ 49us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:44.799166 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.799213 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:44.799329 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.810402 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.810436 11994 leader_election.cc:302] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:44.809055 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.811269 12251 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
W20260808 20:14:44.788318 15852 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.624s user 0.010s sys 0.014s
I20260808 20:14:44.789928 15653 raft_consensus.cc:2726] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260808 20:14:44.801539 15713 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.633s user 0.007s sys 0.022s
I20260808 20:14:44.821491 15873 raft_consensus.cc:2726] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:44.824126 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:44.788007 15592 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:44.833653 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:44.824851 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 369817us
I20260808 20:14:44.824465 12150 raft_consensus.cc:1226] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 6. Current term is 7. Ops: []
I20260808 20:14:44.834774 12150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 348) took 1783 ms. Trace:
I20260808 20:14:44.805565 15858 raft_consensus.cc:2781] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:44.834472 12249 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:44.809329 15699 raft_consensus.cc:2726] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:44.818953 15762 raft_consensus.cc:2781] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:44.835151 12150 rpcz_store.cc:276] 0808 20:14:43.052910 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.808116 (+1755206us) service_pool.cc:227] Handling call
0808 20:14:44.824304 (+ 16188us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.834525 (+ 10221us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.834712 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.813493 15908 raft_consensus.cc:2726] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:44.837947 12251 raft_consensus.cc:1226] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260808 20:14:44.845849 12245 raft_consensus.cc:1226] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:44.853565 12251 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1767) took 1486 ms. Trace:
I20260808 20:14:44.854286 12245 raft_consensus.cc:1226] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:44.855501 12245 raft_consensus.cc:1226] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260808 20:14:44.849323 12150 raft_consensus.cc:1261] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260808 20:14:44.762279 12141 rpcz_store.cc:269] 0808 20:14:43.054967 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.272799 (+1217832us) service_pool.cc:227] Handling call
0808 20:14:44.549385 (+276586us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.856575 12254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1918) took 1873 ms. Trace:
I20260808 20:14:44.856853 12150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 352) took 1802 ms. Trace:
I20260808 20:14:44.858805 12150 rpcz_store.cc:276] 0808 20:14:43.054163 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.848652 (+1794489us) service_pool.cc:227] Handling call
0808 20:14:44.848998 (+ 346us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.856599 (+ 7601us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.856800 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.860307 12150 raft_consensus.cc:1226] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:44.860985 12150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1402) took 1805 ms. Trace:
I20260808 20:14:44.835909 15713 consensus_queue.cc:1225] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer 4cd46f13bceb4c3c9ef71e755e17b995 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260808 20:14:44.861358 12150 rpcz_store.cc:276] 0808 20:14:43.055162 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.859862 (+1804700us) service_pool.cc:227] Handling call
0808 20:14:44.860145 (+ 283us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.860745 (+ 600us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.860936 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.728861 12243 rpcz_store.cc:276] 0808 20:14:42.896256 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.282365 (+1386109us) service_pool.cc:227] Handling call
0808 20:14:44.427280 (+144915us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.511162 (+ 83882us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.521037 (+ 9875us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:44.528990 (+ 7953us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:44.533754 (+ 4764us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.539337 (+ 5583us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:44.555864 (+ 16527us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.853942 12251 rpcz_store.cc:276] 0808 20:14:43.366919 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.811944 (+1445025us) service_pool.cc:227] Handling call
0808 20:14:44.837683 (+ 25739us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:44.853280 (+ 15597us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.853502 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:44.863025 12249 raft_consensus.cc:1226] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:44.819377 15690 consensus_queue.cc:1046] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260808 20:14:44.840831 15602 raft_consensus.cc:2781] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:44.839502 15858 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:44.863137 12150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 358) took 1806 ms. Trace:
I20260808 20:14:44.862598 15906 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:44.839242 15916 raft_consensus.cc:2726] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:44.869168 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:44.869771 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:44.870024 15733 raft_consensus.cc:2726] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:44.870461 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:44.871539 12251 raft_consensus.cc:1226] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
W20260808 20:14:44.875945 15646 consensus_meta.cc:287] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.637s user 0.005s sys 0.001s
I20260808 20:14:44.877470 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:44.877890 15726 leader_election.cc:288] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:44.848177 15762 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:44.843459 15592 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:44.878659 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:45.200361 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:45.079370 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:45.061554 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:45.199718 15602 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:14:45.188503 15918 log.cc:924] Time spent T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.363s user 0.001s sys 0.000s
W20260808 20:14:45.198171 15920 log.cc:924] Time spent T ddb83ac6240244be967995e7771a8b64 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.168s user 0.001s sys 0.000s
I20260808 20:14:44.953620 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:44.863469 12254 rpcz_store.cc:276] 0808 20:14:42.983490 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.312758 (+1329268us) service_pool.cc:227] Handling call
0808 20:14:44.313045 (+ 287us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.313240 (+ 195us) raft_consensus.cc:1520] Early marking committed up to index 3
0808 20:14:44.577070 (+263830us) log.cc:842] Serialized 14 byte log entry
0808 20:14:44.855699 (+278629us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:44.856046 (+ 347us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:44.856144 (+ 98us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.856273 (+ 129us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:44.856467 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10818,"prepare.queue_time_us":483170,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":173,"thread_start_us":503038,"threads_started":2}
I20260808 20:14:45.205605 12152 raft_consensus.cc:3037] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:45.203944 12045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 520) took 1076 ms. Trace:
I20260808 20:14:44.989537 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:45.009363 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:45.078613 13016 leader_election.cc:334] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 2.113s (SENT)
I20260808 20:14:45.009037 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:45.126106 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:45.405571 13073 raft_consensus.cc:3037] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:14:45.270015 15771 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.022s sys 0.009s
W20260808 20:14:45.406949 15771 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.655s user 0.029s sys 0.009s
W20260808 20:14:44.923699 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 228021us
W20260808 20:14:45.209789 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 134672us
I20260808 20:14:45.409605 13014 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
W20260808 20:14:45.016546 12206 connection.cc:232] RPC call timeout handler was delayed by -1.2578872309532017s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1603s for a response
I20260808 20:14:45.206028 12150 rpcz_store.cc:276] 0808 20:14:43.057224 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.862427 (+1805203us) service_pool.cc:227] Handling call
0808 20:14:44.862724 (+ 297us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.862874 (+ 150us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.862897 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:44.862931 (+ 34us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:44.862958 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.863002 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:44.863093 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:45.410499 13014 leader_election.cc:302] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
W20260808 20:14:45.270165 12043 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.752s user 0.006s sys 0.026s
W20260808 20:14:45.410302 12206 connection.cc:232] RPC call timeout handler was delayed by -1.2335872310213745s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.163s for a response
I20260808 20:14:45.408507 13015 leader_election.cc:302] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:45.223084 12137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1404) took 2161 ms. Trace:
W20260808 20:14:45.208200 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 411441us
W20260808 20:14:45.411872 12206 connection.cc:232] RPC call timeout handler was delayed by -1.2020872309803963s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1665s for a response
W20260808 20:14:45.413066 12206 connection.cc:232] RPC call timeout handler was delayed by -1.1453872309066355s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1728s for a response
I20260808 20:14:45.409297 12045 rpcz_store.cc:276] 0808 20:14:44.127311 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.681680 (+554369us) service_pool.cc:227] Handling call
0808 20:14:44.702216 (+ 20536us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:44.703721 (+ 1505us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.703753 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:44.704125 (+ 372us) log.cc:842] Serialized 26 byte log entry
0808 20:14:44.844365 (+140240us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:44.844397 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.844456 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:45.203732 (+359276us) raft_consensus.cc:1671] finished
0808 20:14:45.203754 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:45.203887 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":139969,"threads_started":1,"wal-append.queue_time_us":410241}
W20260808 20:14:45.412272 12043 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.894s user 0.009s sys 0.029s
W20260808 20:14:45.267838 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 387433us
I20260808 20:14:45.268007 12251 raft_consensus.cc:1226] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6. Ops: []
W20260808 20:14:44.866067 15900 log.cc:924] Time spent T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.307s user 0.001s sys 0.000s
I20260808 20:14:45.409034 13016 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:45.215780 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:45.414885 12047 raft_consensus.cc:2354] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5.
I20260808 20:14:45.413189 12102 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:45.415753 12205 connection.cc:232] RPC call timeout handler was delayed by -1.2226956877857447s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3294s for a response
W20260808 20:14:45.415560 12206 connection.cc:232] RPC call timeout handler was delayed by -1.0625872309319675s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18200000000000002s for a response
I20260808 20:14:45.416301 12137 rpcz_store.cc:276] 0808 20:14:43.061368 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.509393 (+1448025us) service_pool.cc:227] Handling call
0808 20:14:44.877318 (+367925us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:45.223018 (+345700us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24654592}
W20260808 20:14:45.457656 12206 connection.cc:232] RPC call timeout handler was delayed by -1.0022872309200466s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1887s for a response
I20260808 20:14:45.417296 12254 raft_consensus.cc:3032] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:45.458271 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:45.458542 12254 raft_consensus.cc:729] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:45.415481 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:45.044770 15926 log.cc:924] Time spent T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.168s user 0.000s sys 0.003s
W20260808 20:14:44.924942 12138 consensus_meta.cc:287] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.683s user 0.009s sys 0.001s
W20260808 20:14:45.007897 15945 log.cc:924] Time spent T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.130s user 0.000s sys 0.002s
I20260808 20:14:45.454859 12251 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1775) took 1090 ms. Trace:
I20260808 20:14:45.468117 12251 rpcz_store.cc:276] 0808 20:14:44.364137 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:45.207738 (+843601us) service_pool.cc:227] Handling call
0808 20:14:45.267820 (+ 60082us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:45.424507 (+156687us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:45.454798 (+ 30291us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:45.469625 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:45.455245 12101 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.24.3:38391 timed out after 2.176s (SENT) [suppressed 70 similar messages]
W20260808 20:14:45.460920 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 46371us
W20260808 20:14:45.454771 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 376177us
W20260808 20:14:45.404027 15924 log.cc:924] Time spent T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.304s user 0.000s sys 0.000s
W20260808 20:14:45.404663 15930 log.cc:924] Time spent T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.279s user 0.000s sys 0.000s
I20260808 20:14:45.414670 15602 raft_consensus.cc:3037] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:45.466241 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:45.320922 15690 raft_consensus.cc:482] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:45.487411 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:45.469740 12251 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1789) took 1101 ms. Trace:
W20260808 20:14:45.460414 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 51991.6us
I20260808 20:14:45.489362 13015 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:45.474673 12138 raft_consensus.cc:2396] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260808 20:14:45.206840 15858 raft_consensus.cc:3037] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:45.076227 15674 raft_consensus.cc:2726] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260808 20:14:45.241078 15650 consensus_meta.cc:287] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.562s user 0.011s sys 0.000s
I20260808 20:14:45.468509 12048 raft_consensus.cc:1261] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260808 20:14:45.290954 15571 consensus_meta.cc:287] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 1.141s user 0.011s sys 0.000s
W20260808 20:14:45.329149 15913 log.cc:924] Time spent T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.463s user 0.000s sys 0.001s
W20260808 20:14:45.329983 15915 log.cc:924] Time spent T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.215s user 0.000s sys 0.001s
I20260808 20:14:45.475569 12205 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:45.475519 12102 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:45.043764 15776 raft_consensus.cc:482] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:45.210143 15762 raft_consensus.cc:3037] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:45.134265 12141 raft_consensus.cc:1261] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 20:14:45.477903 12105 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:45.407893 15891 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:45.479915 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:45.465827 12206 leader_election.cc:302] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:45.481112 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:45.466850 11994 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:44.894666 15624 leader_election.cc:288] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:45.481065 12147 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.000s sys 0.028s
W20260808 20:14:45.486387 12248 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.000s sys 0.024s
I20260808 20:14:45.470722 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:45.488188 12251 rpcz_store.cc:276] 0808 20:14:44.367800 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:45.468857 (+1101057us) service_pool.cc:227] Handling call
0808 20:14:45.469163 (+ 306us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:45.469451 (+ 288us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:45.469475 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:45.469515 (+ 40us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:45.469543 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:45.469591 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:45.469689 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:45.478248 12254 consensus_queue.cc:260] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:45.488843 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:45.471377 15923 raft_consensus.cc:2726] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260808 20:14:45.609040 12248 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s user 0.001s sys 0.025s
I20260808 20:14:45.506316 15826 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:45.502131 12152 raft_consensus.cc:2454] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:14:45.460927 15864 raft_consensus.cc:2781] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:45.481225 15934 raft_consensus.cc:2781] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:45.488749 15771 raft_consensus.cc:3032] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 20:14:45.633082 12244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1930) took 2572 ms. Trace:
I20260808 20:14:45.543610 15916 consensus_queue.cc:1035] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:45.633788 12249 raft_consensus.cc:1203] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[4.3-4.3] Dedup: 4.3->[]
I20260808 20:14:45.634296 12249 raft_consensus.cc:1226] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260808 20:14:45.544761 12250 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1928) took 2501 ms. Trace:
I20260808 20:14:45.634742 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:45.634943 12250 rpcz_store.cc:276] 0808 20:14:43.042940 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.345322 (+1302382us) service_pool.cc:227] Handling call
0808 20:14:44.680961 (+335639us) spinlock_profiling.cc:243] Waited 6.95 ms on lock 0x7b58001a1398. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:44.681172 (+ 211us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:44.694173 (+ 13001us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.694207 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:44.722448 (+ 28241us) log.cc:842] Serialized 26 byte log entry
0808 20:14:45.100926 (+378478us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:45.100965 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:45.101029 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:45.544464 (+443435us) raft_consensus.cc:1671] finished
0808 20:14:45.544484 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:45.544700 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12617,"raft.queue_time_us":12934,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":20308,"spinlock_wait_cycles":15980800,"thread_start_us":338220,"threads_started":1,"wal-append.queue_time_us":614383}
I20260808 20:14:45.500818 12243 raft_consensus.cc:1261] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
W20260808 20:14:45.554448 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 79003us
I20260808 20:14:45.416095 15906 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:45.486577 15646 consensus_queue.cc:1046] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.064s
I20260808 20:14:45.486846 15733 consensus_queue.cc:1035] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
W20260808 20:14:45.558495 12138 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1394) took 2510 ms (client timeout 1630 ms). Trace:
W20260808 20:14:45.607296 12147 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.713s user 0.000s sys 0.033s
I20260808 20:14:45.607962 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:45.506773 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:45.613229 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 146453us
I20260808 20:14:45.614635 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:45.615082 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 131517us
I20260808 20:14:45.615170 12137 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:45.267330 13071 raft_consensus.cc:2454] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
W20260808 20:14:45.617863 12152 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1403) took 2558 ms (client timeout 3294 ms). Trace:
I20260808 20:14:45.618950 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:45.619630 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1945) took 2296 ms. Trace:
I20260808 20:14:45.544777 15911 raft_consensus.cc:2781] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:45.633472 12244 rpcz_store.cc:276] 0808 20:14:43.060397 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.360682 (+1300285us) service_pool.cc:227] Handling call
0808 20:14:44.361302 (+ 620us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 20:14:44.361650 (+ 348us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.361675 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 20:14:44.369554 (+ 7879us) log.cc:842] Serialized 52 byte log entry
0808 20:14:44.570118 (+200564us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:14:44.570158 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.570224 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:45.632842 (+1062618us) raft_consensus.cc:1671] finished
0808 20:14:45.632862 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:45.633026 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":487,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":121,"thread_start_us":200177,"threads_started":1,"wal-append.queue_time_us":584086}
W20260808 20:14:45.659183 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 151693us
I20260808 20:14:45.641264 12245 raft_consensus.cc:1203] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[3.3-3.3] Dedup: 3.3->[]
I20260808 20:14:45.641963 12243 pending_rounds.cc:76] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a: Aborting all ops after (but not including) 6
I20260808 20:14:45.660477 12038 rpcz_store.cc:276] 0808 20:14:43.323123 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.522157 (+1199034us) service_pool.cc:227] Handling call
0808 20:14:44.522493 (+ 336us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:44.522884 (+ 391us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.522912 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:44.523341 (+ 429us) log.cc:842] Serialized 26 byte log entry
0808 20:14:44.806664 (+283323us) raft_consensus.cc:1631] Marking committed up to 1
0808 20:14:44.806706 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.806774 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:45.619412 (+812638us) raft_consensus.cc:1671] finished
0808 20:14:45.619435 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:45.619567 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11705,"raft.queue_time_us":86,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":4116,"thread_start_us":233449,"threads_started":1,"wal-append.queue_time_us":474223}
I20260808 20:14:45.606725 12254 raft_consensus.cc:3037] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:45.493819 15690 raft_consensus.cc:504] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:45.668354 15906 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:45.668519 15971 raft_consensus.cc:2781] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:45.668962 15971 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:45.669358 15971 raft_consensus.cc:3037] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:45.659955 15911 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:45.671320 15911 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:45.637414 15771 raft_consensus.cc:729] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:45.662659 12243 pending_rounds.cc:97] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260808 20:14:45.644879 12141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1681) took 2583 ms. Trace:
I20260808 20:14:45.674124 15967 raft_consensus.cc:2726] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:45.674046 15592 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:45.609506 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:45.673730 12248 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 443) took 1540 ms. Trace:
I20260808 20:14:45.615686 15726 raft_consensus.cc:2726] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260808 20:14:45.633709 15945 log.cc:924] Time spent T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260808 20:14:45.676010 12141 rpcz_store.cc:276] 0808 20:14:43.057501 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.878866 (+1821365us) service_pool.cc:227] Handling call
0808 20:14:45.061784 (+182918us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:45.613070 (+551286us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:45.641111 (+ 28041us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:45.676703 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1399) took 2635 ms. Trace:
I20260808 20:14:45.677066 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:45.677965 12141 raft_consensus.cc:2354] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7.
W20260808 20:14:45.678314 15944 log.cc:924] Time spent T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20260808 20:14:45.678731 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:45.680035 12254 raft_consensus.cc:1261] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 20:14:45.680653 12254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1782) took 1314 ms. Trace:
I20260808 20:14:45.620483 15624 raft_consensus.cc:2781] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260808 20:14:45.651238 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 185378us
W20260808 20:14:45.659297 12152 rpcz_store.cc:269] 0808 20:14:43.059391 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.518666 (+1459275us) service_pool.cc:227] Handling call
0808 20:14:45.200200 (+681534us) spinlock_profiling.cc:243] Waited 1.49 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:45.617725 (+417525us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3416448}
I20260808 20:14:45.701321 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:45.571388 15949 raft_consensus.cc:2726] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:45.660991 12245 raft_consensus.cc:1226] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260808 20:14:45.649899 15699 consensus_queue.cc:1035] T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
I20260808 20:14:45.699075 15858 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:45.549139 15650 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:45.664168 12251 raft_consensus.cc:1226] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 8. Current term is 9. Ops: []
I20260808 20:14:45.663751 15948 consensus_queue.cc:1035] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.041s
I20260808 20:14:45.665849 12244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:44376 (request call id 692) took 3617 ms. Trace:
I20260808 20:14:45.659108 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:45.630141 15864 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:45.675717 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:45.719012 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:45.674203 12243 raft_consensus.cc:2866] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:45.680940 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1454) took 2795 ms. Trace:
I20260808 20:14:45.725001 12039 rpcz_store.cc:276] 0808 20:14:42.886274 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.396554 (+1510280us) service_pool.cc:227] Handling call
0808 20:14:44.510245 (+113691us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 20:14:44.536836 (+ 26591us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.536865 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 20:14:44.550119 (+ 13254us) log.cc:842] Serialized 52 byte log entry
0808 20:14:44.845420 (+295301us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:44.845455 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.845514 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:45.680719 (+835205us) raft_consensus.cc:1671] finished
0808 20:14:45.680738 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:45.680865 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7486,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":130724,"thread_start_us":123180,"threads_started":1,"wal-append.queue_time_us":949847}
I20260808 20:14:45.682209 15646 raft_consensus.cc:3032] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:45.668876 15690 leader_election.cc:288] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:45.668923 15906 consensus_queue.cc:1046] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
I20260808 20:14:45.629444 15934 raft_consensus.cc:482] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:45.651898 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:45.682407 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 523) took 1127 ms. Trace:
I20260808 20:14:45.633468 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:45.731477 13076 raft_consensus.cc:2379] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:45.675532 12248 rpcz_store.cc:276] 0808 20:14:44.133436 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.814669 (+681233us) service_pool.cc:227] Handling call
0808 20:14:44.842647 (+ 27978us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:44.842886 (+ 239us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.842913 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:44.850007 (+ 7094us) log.cc:842] Serialized 26 byte log entry
0808 20:14:45.609452 (+759445us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:45.609494 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:45.609554 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:45.673551 (+ 63997us) raft_consensus.cc:1671] finished
0808 20:14:45.673569 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:45.673679 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":685421,"threads_started":1,"wal-append.queue_time_us":794911}
W20260808 20:14:45.718752 15938 log.cc:924] Time spent T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.298s user 0.000s sys 0.001s
I20260808 20:14:45.677067 12146 rpcz_store.cc:276] 0808 20:14:43.041692 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.685722 (+1644030us) service_pool.cc:227] Handling call
0808 20:14:44.718771 (+ 33049us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:44.719003 (+ 232us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.719026 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:44.719434 (+ 408us) log.cc:842] Serialized 26 byte log entry
0808 20:14:44.822034 (+102600us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:44.822345 (+ 311us) log.cc:842] Serialized 14 byte log entry
0808 20:14:45.062368 (+240023us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:45.062454 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:45.676516 (+614062us) raft_consensus.cc:1671] finished
0808 20:14:45.676533 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:45.676650 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":857,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":51868,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":107,"thread_start_us":273135,"threads_started":2,"wal-append.queue_time_us":125958}
I20260808 20:14:45.675658 15107 raft_consensus.cc:3032] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 20:14:45.572827 15776 raft_consensus.cc:504] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:45.698374 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 222886us
W20260808 20:14:45.685367 15940 log.cc:924] Time spent T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.178s user 0.000s sys 0.000s
I20260808 20:14:45.698525 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1932) took 2672 ms. Trace:
I20260808 20:14:45.652156 15951 raft_consensus.cc:2726] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260808 20:14:45.649600 12138 rpcz_store.cc:269] 0808 20:14:43.040326 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.041604 (+ 1278us) service_pool.cc:227] Handling call
0808 20:14:45.551057 (+2509453us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:45.744542 12036 rpcz_store.cc:276] 0808 20:14:43.026317 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.518674 (+1492357us) service_pool.cc:227] Handling call
0808 20:14:44.532617 (+ 13943us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:44.537009 (+ 4392us) raft_consensus.cc:1520] Early marking committed up to index 2
0808 20:14:44.678861 (+141852us) log.cc:842] Serialized 14 byte log entry
0808 20:14:45.029855 (+350994us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:45.157031 (+127176us) log.cc:842] Serialized 26 byte log entry
0808 20:14:45.157179 (+ 148us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:45.175246 (+ 18067us) log.cc:842] Serialized 14 byte log entry
0808 20:14:45.175519 (+ 273us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:45.175593 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:45.186928 (+ 11335us) spinlock_profiling.cc:243] Waited 4.94 ms on lock 0x7b48000b01d8. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96d1817f1 00007fa96d1788e7 00007fa96d17877a 00007fa96d26c95f 00007fa96d26c627 00007fa96d280b2c 00007fa96d280ad8 00007fa96d280a58 00007fa96d280a13 00007fa96d27fcbc 00007fa96c201be6
0808 20:14:45.698182 (+511254us) raft_consensus.cc:1671] finished
0808 20:14:45.698237 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:45.698443 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3949,"prepare.queue_time_us":159638,"prepare.run_cpu_time_us":6937,"prepare.run_wall_time_us":20414,"raft.queue_time_us":16430,"raft.run_cpu_time_us":3025,"raft.run_wall_time_us":667391,"spinlock_wait_cycles":11357056,"thread_start_us":279356,"threads_started":1,"wal-append.queue_time_us":955220}
I20260808 20:14:45.700295 12254 rpcz_store.cc:276] 0808 20:14:44.366028 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:45.408025 (+1041997us) service_pool.cc:227] Handling call
0808 20:14:45.417019 (+ 8994us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:45.680411 (+263392us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:45.680597 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:45.699019 13080 raft_consensus.cc:2379] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:45.701953 12152 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:14:45.750595 13072 raft_consensus.cc:2379] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:45.751484 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:45.752058 13072 raft_consensus.cc:2454] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:45.753849 12138 raft_consensus.cc:1261] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 20:14:45.754537 13073 raft_consensus.cc:2454] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:45.757615 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 519) took 1630 ms. Trace:
I20260808 20:14:45.757951 12040 rpcz_store.cc:276] 0808 20:14:44.127050 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.553596 (+426546us) service_pool.cc:227] Handling call
0808 20:14:44.553867 (+ 271us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:44.695577 (+141710us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.695607 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:44.696008 (+ 401us) log.cc:842] Serialized 26 byte log entry
0808 20:14:45.078326 (+382318us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:45.078363 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:45.078425 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:45.757445 (+679020us) raft_consensus.cc:1671] finished
0808 20:14:45.757461 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:45.757565 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":328163,"threads_started":1,"wal-append.queue_time_us":761322}
I20260808 20:14:45.760680 15776 leader_election.cc:288] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:45.744110 15107 raft_consensus.cc:729] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:45.761381 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:45.761806 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:45.762632 12142 raft_consensus.cc:2379] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 7: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
W20260808 20:14:45.762993 12142 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 353) took 2708 ms (client timeout 3092 ms). Trace:
W20260808 20:14:45.763319 12142 rpcz_store.cc:269] 0808 20:14:43.054446 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.513618 (+1459172us) service_pool.cc:227] Handling call
0808 20:14:45.761624 (+1248006us) spinlock_profiling.cc:243] Waited 6.54 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:45.762939 (+ 1315us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15043968}
I20260808 20:14:45.736353 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:45.765275 12046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 522) took 1532 ms. Trace:
I20260808 20:14:45.765652 12046 rpcz_store.cc:276] 0808 20:14:44.233284 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.685010 (+451726us) service_pool.cc:227] Handling call
0808 20:14:44.701134 (+ 16124us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:44.701354 (+ 220us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.701381 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:44.721821 (+ 20440us) log.cc:842] Serialized 26 byte log entry
0808 20:14:44.864417 (+142596us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:44.864448 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:44.864513 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:45.765050 (+900537us) raft_consensus.cc:1671] finished
0808 20:14:45.765069 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:45.765206 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":110723,"threads_started":1,"wal-append.queue_time_us":454138}
I20260808 20:14:45.767418 12147 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 342) took 2720 ms. Trace:
I20260808 20:14:45.768049 15667 raft_consensus.cc:3032] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:45.768471 15667 raft_consensus.cc:729] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:45.768829 15762 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:45.769223 15667 consensus_queue.cc:260] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:45.770500 15667 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:45.773191 15650 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:45.773888 15762 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:45.705210 12206 leader_election.cc:302] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:45.775107 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 69970.8us
I20260808 20:14:45.775501 15743 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:45.775981 15743 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:45.776392 12206 leader_election.cc:302] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:45.777163 12144 raft_consensus.cc:2396] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260808 20:14:45.732565 15934 raft_consensus.cc:3037] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:45.700701 15624 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:45.778262 15624 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:45.551541 15571 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:45.705858 15941 consensus_queue.cc:1046] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.075s
I20260808 20:14:45.667084 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:45.781569 12137 raft_consensus.cc:3037] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:45.781843 13079 raft_consensus.cc:2379] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 5: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:45.781566 15592 raft_consensus.cc:2726] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:45.777751 15969 raft_consensus.cc:2726] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 20:14:45.719141 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 74076.4us
I20260808 20:14:45.712576 12043 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 1918) took 3183 ms. Trace:
I20260808 20:14:45.785455 12043 rpcz_store.cc:276] 0808 20:14:42.529123 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.338904 (+1809781us) service_pool.cc:227] Handling call
0808 20:14:44.393316 (+ 54412us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:44.393660 (+ 344us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.393692 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:44.394113 (+ 421us) log.cc:842] Serialized 26 byte log entry
0808 20:14:45.414565 (+1020452us) raft_consensus.cc:1631] Marking committed up to 1
0808 20:14:45.414620 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:45.414682 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:45.712356 (+297674us) raft_consensus.cc:1671] finished
0808 20:14:45.712378 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:45.712515 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":33183,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":76086,"spinlock_wait_cycles":1354496,"thread_start_us":896360,"threads_started":1,"wal-append.queue_time_us":1198943}
I20260808 20:14:45.709903 12244 rpcz_store.cc:276] 0808 20:14:42.048260 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:45.664858 (+3616598us) service_pool.cc:227] Handling call
0808 20:14:45.665796 (+ 938us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:45.788048 12245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1772) took 1499 ms. Trace:
I20260808 20:14:45.788331 12137 raft_consensus.cc:2454] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:14:45.767786 12147 rpcz_store.cc:276] 0808 20:14:43.047328 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.763318 (+1715990us) service_pool.cc:227] Handling call
0808 20:14:44.809734 (+ 46416us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:44.819259 (+ 9525us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.819291 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:44.846883 (+ 27592us) log.cc:842] Serialized 26 byte log entry
0808 20:14:45.650425 (+803542us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:45.650465 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:45.650542 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:45.767207 (+116665us) raft_consensus.cc:1671] finished
0808 20:14:45.767227 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:45.767359 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":911,"raft.queue_time_us":42493,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":5235,"thread_start_us":756194,"threads_started":1,"wal-append.queue_time_us":759421}
W20260808 20:14:45.782286 12001 leader_election.cc:334] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Timed out: RequestConsensusVote RPC to 127.10.24.4:36901 timed out after 1.657s (SENT)
I20260808 20:14:45.781638 15941 raft_consensus.cc:3032] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 7 LEADER]: Stepping down as leader of term 7
I20260808 20:14:45.791664 15941 raft_consensus.cc:729] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:45.727695 12150 raft_consensus.cc:1261] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 20:14:45.698616 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:45.707474 12241 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1765) took 2644 ms (client timeout 2155 ms). Trace:
W20260808 20:14:45.805004 12241 rpcz_store.cc:269] 0808 20:14:43.063179 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:45.707287 (+2644108us) service_pool.cc:214] Skipping call since client already timed out
0808 20:14:45.707396 (+ 109us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 20:14:45.805666 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:45.806031 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:45.806823 12241 raft_consensus.cc:2379] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 7: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:45.807642 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:45.706634 15964 log.cc:924] Time spent T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260808 20:14:45.707808 15858 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:45.809684 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:45.810562 15322 raft_consensus.cc:3032] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 20:14:45.810951 15322 raft_consensus.cc:729] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:45.811818 15322 consensus_queue.cc:260] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:45.813233 15886 consensus_queue.cc:1035] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:14:45.710762 15864 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:45.792431 15941 consensus_queue.cc:260] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:45.821671 12254 raft_consensus.cc:2379] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:45.733094 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:45.705976 15142 consensus_queue.cc:1046] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260808 20:14:45.728036 15646 raft_consensus.cc:729] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
W20260808 20:14:45.738123 15675 consensus_meta.cc:287] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 1.445s user 0.010s sys 0.000s
W20260808 20:14:45.742331 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 75955.7us
I20260808 20:14:45.839581 13077 raft_consensus.cc:1226] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260808 20:14:45.840044 12144 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1387) took 2807 ms (client timeout 3292 ms). Trace:
I20260808 20:14:45.650681 12247 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 1936) took 2285 ms. Trace:
I20260808 20:14:45.841616 12146 raft_consensus.cc:1261] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260808 20:14:45.841929 12146 pending_rounds.cc:76] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9: Aborting all ops after (but not including) 4
I20260808 20:14:45.842113 12146 pending_rounds.cc:97] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260808 20:14:45.842406 12146 raft_consensus.cc:2866] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 20:14:45.732585 15614 consensus_meta.cc:287] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.970s user 0.010s sys 0.000s
I20260808 20:14:45.673596 15771 consensus_queue.cc:260] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:45.744728 12205 leader_election.cc:302] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:45.704911 11994 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
W20260808 20:14:45.850486 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 151770us
I20260808 20:14:45.851147 11994 leader_election.cc:302] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:45.781004 15571 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:45.788404 12245 rpcz_store.cc:276] 0808 20:14:44.289084 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.858213 (+569129us) service_pool.cc:227] Handling call
0808 20:14:45.641146 (+782933us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:45.704373 (+ 63227us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:45.787871 (+ 83498us) spinlock_profiling.cc:243] Waited 783 ms on lock 0x7b5400060250. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d9a09 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:45.787993 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1799942400}
I20260808 20:14:45.794059 12150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1684) took 2731 ms. Trace:
I20260808 20:14:45.705822 12105 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:45.859997 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 115346us
I20260808 20:14:45.799239 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:45.799896 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:45.801397 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:45.860628 12205 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:45.860929 12248 raft_consensus.cc:3037] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:45.781895 12243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1778) took 1416 ms. Trace:
I20260808 20:14:45.802156 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:45.802212 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:45.804214 13016 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:45.836854 15322 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:45.813463 15743 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:45.729544 12041 rpcz_store.cc:276] 0808 20:14:44.555427 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.687236 (+131809us) service_pool.cc:227] Handling call
0808 20:14:44.687536 (+ 300us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:44.688195 (+ 659us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:44.688223 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:44.697378 (+ 9155us) log.cc:842] Serialized 26 byte log entry
0808 20:14:45.076224 (+378846us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:45.076262 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:45.076327 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:45.682220 (+605893us) raft_consensus.cc:1671] finished
0808 20:14:45.682240 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:45.682355 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5615,"raft.queue_time_us":356,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":7272,"thread_start_us":317411,"threads_started":1,"wal-append.queue_time_us":727664}
I20260808 20:14:45.825752 13078 raft_consensus.cc:3037] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:45.868335 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:45.852782 15638 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:45.833678 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:45.840452 12144 rpcz_store.cc:269] 0808 20:14:43.032669 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:43.037471 (+ 4802us) service_pool.cc:227] Handling call
0808 20:14:43.436683 (+399212us) spinlock_profiling.cc:243] Waited 2.44 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:45.839873 (+2403190us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b54001801d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:45.839991 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3761200256}
I20260808 20:14:45.869334 15638 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:45.869978 12241 raft_consensus.cc:2379] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 3: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:45.839411 15666 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:45.870368 15666 raft_consensus.cc:3037] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:45.870796 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:45.871570 12241 raft_consensus.cc:2379] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 6: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:45.872359 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:45.846750 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:45.873158 12140 raft_consensus.cc:2454] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 2.
W20260808 20:14:45.873581 12140 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 343) took 2826 ms (client timeout 3201 ms). Trace:
I20260808 20:14:45.874089 15969 raft_consensus.cc:2781] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:45.874516 15969 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:45.874858 15969 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:45.851992 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:45.875535 12045 raft_consensus.cc:3037] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
W20260808 20:14:45.873898 12140 rpcz_store.cc:269] 0808 20:14:43.047563 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.516303 (+1468740us) service_pool.cc:227] Handling call
0808 20:14:45.846577 (+1330274us) spinlock_profiling.cc:243] Waited 3.78 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:45.873531 (+ 26954us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8683904}
I20260808 20:14:45.876871 15646 consensus_queue.cc:260] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:45.854023 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:45.857911 12249 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1774) took 1494 ms. Trace:
I20260808 20:14:45.869479 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:45.877956 15971 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:45.878611 13071 raft_consensus.cc:2379] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:45.858081 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:45.877960 15646 raft_consensus.cc:3037] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:45.862025 12205 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:45.860636 12250 raft_consensus.cc:3037] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 7
I20260808 20:14:45.863782 12142 raft_consensus.cc:1226] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260808 20:14:45.863693 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:45.881799 12251 raft_consensus.cc:2379] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 6: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
W20260808 20:14:45.882447 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 78333.7us
I20260808 20:14:45.883042 15968 raft_consensus.cc:2726] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:45.883425 13016 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:45.859935 12150 rpcz_store.cc:276] 0808 20:14:43.062258 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:45.727070 (+2664812us) service_pool.cc:227] Handling call
0808 20:14:45.727376 (+ 306us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:45.793815 (+ 66439us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:45.794006 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:45.864405 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 158640us
I20260808 20:14:45.885496 12254 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:45.865272 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:45.866998 15941 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:45.867244 13079 raft_consensus.cc:1226] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 LEARNER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260808 20:14:45.807930 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:45.841080 12247 rpcz_store.cc:276] 0808 20:14:43.364186 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.805600 (+1441414us) service_pool.cc:227] Handling call
0808 20:14:44.829406 (+ 23806us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:44.829641 (+ 235us) raft_consensus.cc:1520] Early marking committed up to index 3
0808 20:14:45.249518 (+419877us) log.cc:842] Serialized 14 byte log entry
0808 20:14:45.640851 (+391333us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:45.640914 (+ 63us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:14:45.640964 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:45.641030 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:45.649857 (+ 8827us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":18461,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":52,"raft.queue_time_us":174571,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":26584,"thread_start_us":794353,"threads_started":3,"wal-append.queue_time_us":417713}
I20260808 20:14:45.897596 12039 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:45.878233 12249 rpcz_store.cc:276] 0808 20:14:44.363782 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:44.870708 (+506926us) service_pool.cc:227] Handling call
0808 20:14:45.633629 (+762921us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:45.634632 (+ 1003us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:45.857742 (+223110us) spinlock_profiling.cc:243] Waited 763 ms on lock 0x7b5400088c50. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d9a09 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:45.857859 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1753800448}
I20260808 20:14:45.876991 15142 raft_consensus.cc:3032] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:45.880623 12205 leader_election.cc:302] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:45.877699 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 100206us
I20260808 20:14:45.866196 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:45.887573 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1409) took 2654 ms. Trace:
I20260808 20:14:45.761420 15602 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:45.886623 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:45.901057 12205 leader_election.cc:302] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:45.888306 12244 raft_consensus.cc:2379] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 9: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:45.901444 12206 leader_election.cc:302] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:45.906008 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:45.906008 12205 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:45.906910 12245 raft_consensus.cc:2379] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 6: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:45.849370 15908 raft_consensus.cc:2781] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:45.849577 15675 raft_consensus.cc:2726] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 20:14:45.903970 12001 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:45.853088 15892 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:45.901594 12146 rpcz_store.cc:276] 0808 20:14:43.232875 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:45.746890 (+2514015us) service_pool.cc:227] Handling call
0808 20:14:45.841305 (+ 94415us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:45.842779 (+ 1474us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:45.887392 (+ 44613us) spinlock_profiling.cc:243] Waited 63.6 ms on lock 0x7b54001801d0. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d9a09 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:45.887522 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":146166656}
I20260808 20:14:45.862917 15886 raft_consensus.cc:2781] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 20:14:45.898243 15771 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:45.886724 12241 raft_consensus.cc:3037] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:45.885819 15776 raft_consensus.cc:2781] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:45.915877 12152 raft_consensus.cc:1226] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:45.918866 12244 raft_consensus.cc:1226] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:45.921990 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:45.922677 13071 raft_consensus.cc:1226] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 7. Current term is 8. Ops: []
I20260808 20:14:45.881693 15650 consensus_queue.cc:1046] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 8.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.027s
I20260808 20:14:45.923581 15971 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:45.900147 15596 raft_consensus.cc:2781] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:45.919766 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:45.898540 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:45.900691 15762 raft_consensus.cc:2726] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:45.929965 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:45.959311 12146 raft_consensus.cc:2454] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8.
I20260808 20:14:45.929566 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:45.934387 12039 raft_consensus.cc:2454] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:45.899992 15142 raft_consensus.cc:729] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:45.904665 15592 raft_consensus.cc:2726] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:45.941752 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:45.944759 15990 consensus_queue.cc:1046] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 20:14:45.913720 15967 raft_consensus.cc:2781] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:45.948997 15596 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:45.957914 12045 raft_consensus.cc:2454] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:14:45.960386 12244 raft_consensus.cc:2379] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 9: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:45.972821 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:45.973657 12137 raft_consensus.cc:2379] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:45.926883 15776 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:45.904834 15835 consensus_queue.cc:1046] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260808 20:14:45.966069 13080 raft_consensus.cc:3037] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:45.929358 15873 raft_consensus.cc:2726] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:45.867267 12243 rpcz_store.cc:276] 0808 20:14:44.365005 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:45.227079 (+862074us) service_pool.cc:227] Handling call
0808 20:14:45.456214 (+229135us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:45.746815 (+290601us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:45.781838 (+ 35023us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:45.910729 15571 consensus_queue.cc:1035] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
W20260808 20:14:45.971792 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 70415.5us
I20260808 20:14:45.900249 15916 consensus_queue.cc:1035] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
W20260808 20:14:45.974960 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 69039.3us
W20260808 20:14:45.975664 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 76209.5us
I20260808 20:14:45.976248 13079 raft_consensus.cc:1226] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:45.977082 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:45.964923 15971 raft_consensus.cc:2781] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:45.979815 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:45.995673 12244 raft_consensus.cc:3037] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:45.981570 12206 leader_election.cc:398] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Vote denied by peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139) with higher term. Message: Invalid argument: T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7.
I20260808 20:14:45.996215 12206 leader_election.cc:401] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260808 20:14:45.992023 12250 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 7.
I20260808 20:14:45.974984 15596 raft_consensus.cc:3037] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:45.995213 12137 raft_consensus.cc:1261] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 20:14:45.968743 15967 raft_consensus.cc:482] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:45.913445 15892 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:45.996699 12205 leader_election.cc:302] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:45.978267 15776 raft_consensus.cc:3037] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:46.017421 13079 raft_consensus.cc:1226] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:46.017938 15997 raft_consensus.cc:3032] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:45.993299 15970 raft_consensus.cc:2726] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:46.026541 15892 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:46.006850 15967 raft_consensus.cc:3037] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:14:46.032198 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 50720.3us
I20260808 20:14:46.015239 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:46.066860 12250 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:14:46.067277 12206 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:46.067777 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:46.069002 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:46.040547 13078 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 7.
W20260808 20:14:46.051946 12207 connection.cc:232] RPC call timeout handler was delayed by -1.6269167608115822s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.167s for a response
W20260808 20:14:46.073202 12207 connection.cc:232] RPC call timeout handler was delayed by -1.4478167607448995s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1869s for a response
I20260808 20:14:46.073841 12207 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:46.034579 12248 raft_consensus.cc:2454] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:46.057224 12101 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:46.064611 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 68068.7us
I20260808 20:14:46.075760 12247 raft_consensus.cc:1226] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:46.075925 13078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 716) took 1015 ms. Trace:
I20260808 20:14:46.076401 13078 rpcz_store.cc:276] 0808 20:14:45.061057 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:45.732840 (+671783us) service_pool.cc:227] Handling call
0808 20:14:46.075865 (+343025us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:46.064954 16014 raft_consensus.cc:2781] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 20:14:46.078327 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:46.079012 12137 raft_consensus.cc:2454] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:46.094962 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:46.098649 12146 raft_consensus.cc:2379] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:46.102074 12206 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:46.053745 15997 raft_consensus.cc:729] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:46.100976 12244 raft_consensus.cc:2454] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:46.103863 12101 leader_election.cc:302] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:46.055264 15971 raft_consensus.cc:686] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:46.020581 15911 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:46.077425 16014 raft_consensus.cc:482] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
W20260808 20:14:46.117568 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 457019us
I20260808 20:14:46.054996 15989 raft_consensus.cc:2726] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139) with higher term. Message: Invalid argument: T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7.
I20260808 20:14:46.121137 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:46.125742 12144 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:14:46.122999 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:46.124101 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 50351.3us
I20260808 20:14:46.094147 15992 raft_consensus.cc:2781] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:46.131681 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:46.095880 15951 raft_consensus.cc:2726] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 20:14:46.135932 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:46.137904 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:46.138680 12142 raft_consensus.cc:2379] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:46.009044 15142 consensus_queue.cc:260] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:46.151237 15990 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:46.126091 15873 raft_consensus.cc:482] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:46.131168 15602 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:46.156725 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:46.159312 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:46.126884 15886 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:46.127946 16014 raft_consensus.cc:3037] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:46.165557 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:46.153338 15989 raft_consensus.cc:2726] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:46.164290 15886 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:14:46.168200 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:46.169638 13076 raft_consensus.cc:2379] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 7: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:46.158239 15990 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:46.171892 15990 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:46.172981 15985 log.cc:924] Time spent T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.166s user 0.001s sys 0.000s
I20260808 20:14:46.152489 15699 raft_consensus.cc:2726] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:46.153167 15992 raft_consensus.cc:686] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:46.165449 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:46.152853 15624 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:46.211817 15984 log.cc:924] Time spent T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.075s user 0.000s sys 0.001s
I20260808 20:14:46.166033 15873 raft_consensus.cc:504] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:46.210667 13075 raft_consensus.cc:1261] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 20:14:46.234766 15638 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:46.234998 16004 log.cc:924] Time spent T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260808 20:14:46.212013 15971 consensus_queue.cc:237] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:46.237200 15107 consensus_queue.cc:260] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:46.253542 15934 raft_consensus.cc:504] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:46.254917 15596 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:46.188159 15997 consensus_queue.cc:260] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:46.259967 15997 raft_consensus.cc:3037] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:46.266327 15864 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:46.267269 15666 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:46.255625 15934 leader_election.cc:288] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:46.270267 15886 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:46.272511 15941 raft_consensus.cc:2726] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:46.275435 16014 raft_consensus.cc:504] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:46.286448 15107 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:14:46.285106 15911 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:46.293277 13079 raft_consensus.cc:2454] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
W20260808 20:14:46.246089 16016 log.cc:924] Time spent T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20260808 20:14:46.323700 16014 leader_election.cc:288] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:46.319005 15992 consensus_queue.cc:237] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:46.361542 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 234368us
I20260808 20:14:46.366837 12105 leader_election.cc:302] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:46.366622 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:46.367362 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:46.367718 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:46.368345 12147 raft_consensus.cc:2362] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8. Re-sending same reply.
I20260808 20:14:46.369369 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:46.369535 12105 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:46.369674 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:46.370553 12147 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:14:46.368160 12152 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:14:46.370946 12105 leader_election.cc:302] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:46.368242 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:46.366518 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:46.372046 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:46.372534 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:46.372560 12142 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:46.373149 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:46.373351 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:46.373693 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:46.373924 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:46.374235 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:46.371642 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:46.374475 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:46.374920 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:46.375276 12141 raft_consensus.cc:2396] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260808 20:14:46.375438 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:46.371655 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:46.378116 15666 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:46.303565 13074 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:14:46.308324 15624 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:46.386659 13073 raft_consensus.cc:1226] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260808 20:14:46.386866 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 284702us
I20260808 20:14:46.392001 12206 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:46.392114 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:46.307701 15596 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:46.394264 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:46.394528 12206 leader_election.cc:302] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:46.395074 15596 consensus_queue.cc:1035] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:46.399148 15666 raft_consensus.cc:482] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:46.400017 15666 raft_consensus.cc:504] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:46.400497 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:46.401319 12146 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
I20260808 20:14:46.401672 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:46.405087 13077 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53290 (request call id 591) took 1330 ms. Trace:
I20260808 20:14:46.378151 12038 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 7.
I20260808 20:14:46.408094 12046 raft_consensus.cc:1226] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:46.409075 12045 raft_consensus.cc:1226] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:46.409551 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:46.410445 12039 raft_consensus.cc:3037] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:46.410669 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:46.411206 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:46.411485 12043 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:46.411962 12038 raft_consensus.cc:3037] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:46.412581 15666 leader_election.cc:288] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:46.417055 15596 raft_consensus.cc:2781] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:46.417512 15596 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:46.417824 15596 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:46.420230 13080 raft_consensus.cc:2454] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:46.421586 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:46.423520 12142 raft_consensus.cc:1226] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:46.424847 15602 consensus_queue.cc:1035] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:14:46.426328 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:46.426712 15602 consensus_queue.cc:1035] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 20:14:46.428526 15602 raft_consensus.cc:2781] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:46.428756 12146 raft_consensus.cc:1226] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 8. Current term is 9. Ops: []
I20260808 20:14:46.429207 15602 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:46.429740 15602 raft_consensus.cc:3037] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:46.430542 12146 raft_consensus.cc:1226] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:46.430799 12147 raft_consensus.cc:1226] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 4. Ops: []
W20260808 20:14:46.430259 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 59530.9us
I20260808 20:14:46.375703 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:46.405463 13077 rpcz_store.cc:276] 0808 20:14:45.074084 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:45.902050 (+827966us) service_pool.cc:227] Handling call
0808 20:14:45.902320 (+ 270us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:45.902496 (+ 176us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 20:14:46.065528 (+163032us) log.cc:842] Serialized 14 byte log entry
0808 20:14:46.404308 (+338780us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:46.404416 (+ 108us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:46.404589 (+ 173us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:46.404691 (+ 102us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:46.404867 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15668,"prepare.queue_time_us":67896,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"thread_start_us":478174,"threads_started":2,"wal-append.queue_time_us":260163}
W20260808 20:14:46.435485 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 70853.2us
I20260808 20:14:46.438180 12144 raft_consensus.cc:2379] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 7: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:46.437981 15624 consensus_queue.cc:1035] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:14:46.370887 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:46.444792 15624 consensus_queue.cc:1046] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:46.446138 15624 raft_consensus.cc:3032] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:46.446614 15624 raft_consensus.cc:729] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:46.447770 15624 consensus_queue.cc:260] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:46.449532 15624 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:46.457176 12045 pending_rounds.cc:76] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting all ops after (but not including) 4
I20260808 20:14:46.457569 12045 pending_rounds.cc:97] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260808 20:14:46.457957 12045 raft_consensus.cc:2866] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:46.457293 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:46.463472 12039 raft_consensus.cc:2454] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:46.458406 15653 consensus_queue.cc:1046] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 20:14:46.477785 12043 raft_consensus.cc:2454] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6.
W20260808 20:14:46.489799 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 52992.8us
I20260808 20:14:46.495055 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:46.496304 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 385124us
W20260808 20:14:46.495500 12249 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.007s sys 0.010s
I20260808 20:14:46.509265 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:46.512719 12249 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.533s user 0.008s sys 0.010s
W20260808 20:14:46.515688 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 119503us
I20260808 20:14:46.527901 12000 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:46.501178 15690 raft_consensus.cc:2726] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:14:46.522023 12046 raft_consensus.cc:2454] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:46.532792 15674 raft_consensus.cc:2781] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:46.533653 15948 raft_consensus.cc:2741] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260808 20:14:46.533092 12039 raft_consensus.cc:3037] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:46.537365 15653 raft_consensus.cc:3032] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:46.537765 15653 raft_consensus.cc:729] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:46.538836 15653 consensus_queue.cc:260] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:46.322712 15969 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:46.533934 11994 leader_election.cc:334] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Timed out: RequestConsensusVote RPC to 127.10.24.3:38391 timed out after 1.583s (SENT)
I20260808 20:14:46.542399 15646 consensus_queue.cc:1046] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260808 20:14:46.541592 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 49913.6us
I20260808 20:14:46.543447 15969 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:46.543725 15646 raft_consensus.cc:3032] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:46.544176 15646 raft_consensus.cc:729] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:46.527760 15941 raft_consensus.cc:2781] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:46.561995 15941 raft_consensus.cc:686] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:46.564548 15596 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
W20260808 20:14:46.565125 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 53663us
W20260808 20:14:46.565786 16029 log.cc:924] Time spent T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.239s user 0.001s sys 0.000s
I20260808 20:14:46.569236 15873 leader_election.cc:288] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:46.570369 12205 leader_election.cc:398] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Vote denied by peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747) with higher term. Message: Invalid argument: T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5.
W20260808 20:14:46.566210 12040 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.007s sys 0.009s
I20260808 20:14:46.574833 15941 consensus_queue.cc:237] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:46.575436 15967 raft_consensus.cc:504] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:46.565263 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 74633.6us
I20260808 20:14:46.566007 15614 raft_consensus.cc:2781] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:14:46.577597 12043 raft_consensus.cc:3037] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:46.596784 15614 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:46.599217 15614 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:46.603394 15602 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:46.605588 12038 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
W20260808 20:14:46.606911 12040 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.686s user 0.008s sys 0.009s
I20260808 20:14:46.608142 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:46.611545 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:46.612787 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 453598us
I20260808 20:14:46.607056 15969 consensus_queue.cc:1035] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.007s
W20260808 20:14:46.614181 12241 consensus_meta.cc:287] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.691s user 0.011s sys 0.007s
I20260808 20:14:46.614638 12241 raft_consensus.cc:2454] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:46.615140 15602 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:46.615954 15969 consensus_queue.cc:1035] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:14:46.617179 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:46.619681 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:46.621513 12101 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:46.614084 12000 leader_election.cc:334] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.624s (SENT)
I20260808 20:14:46.623090 12101 leader_election.cc:302] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:46.624631 12101 leader_election.cc:302] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:46.613731 12207 leader_election.cc:302] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:46.631909 12043 raft_consensus.cc:2454] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:46.620185 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:46.635489 12140 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:14:46.620852 15596 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:46.630663 15602 raft_consensus.cc:2726] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:46.640985 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:46.646379 15743 raft_consensus.cc:2726] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:46.634536 15646 consensus_queue.cc:260] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:46.650974 15646 raft_consensus.cc:3037] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:46.653643 15949 raft_consensus.cc:2726] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:46.656456 12036 raft_consensus.cc:1261] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 20:14:46.657512 15864 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:46.661087 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:46.662489 11872 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet cb8b79bb1c59483e93af813653a73a31 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica b9d94f35ea15421aa1d721201c782d8a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:46.669164 13014 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:46.671267 15596 raft_consensus.cc:2726] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:46.671188 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:46.672549 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:46.673355 12150 raft_consensus.cc:3037] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:46.636003 15674 raft_consensus.cc:686] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:46.678330 13071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53290 (request call id 589) took 1604 ms. Trace:
I20260808 20:14:46.678076 15674 consensus_queue.cc:237] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:46.678794 13071 rpcz_store.cc:276] 0808 20:14:45.073572 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:45.880079 (+806507us) service_pool.cc:227] Handling call
0808 20:14:45.880352 (+ 273us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:45.948923 (+ 68571us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:46.677857 (+728934us) spinlock_profiling.cc:243] Waited 32.1 ms on lock 0x7b5400711bd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:46.678120 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":73895680}
I20260808 20:14:46.681339 15923 raft_consensus.cc:2781] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:46.681918 15923 raft_consensus.cc:686] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:46.643738 15967 leader_election.cc:288] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:46.642731 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:46.683187 15923 consensus_queue.cc:237] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:46.684177 15614 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:46.671463 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 57857us
I20260808 20:14:46.678682 12000 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:46.689872 12039 raft_consensus.cc:2454] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6.
I20260808 20:14:46.705876 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:46.709991 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 272776us
I20260808 20:14:46.710731 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:46.713948 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:46.717947 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:46.720072 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:46.722244 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:46.722949 12241 raft_consensus.cc:2354] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5.
W20260808 20:14:46.723606 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 70844.4us
I20260808 20:14:46.726380 12102 leader_election.cc:302] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
W20260808 20:14:46.727514 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 48981.9us
I20260808 20:14:46.727931 12102 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:46.728826 16064 raft_consensus.cc:2781] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:46.729831 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:46.730809 12248 raft_consensus.cc:2396] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 20:14:46.731577 15997 raft_consensus.cc:2726] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:46.733563 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:46.734817 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:46.735558 12041 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:46.737038 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:46.738415 12250 raft_consensus.cc:1226] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260808 20:14:46.738632 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:46.739852 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:46.740715 12249 raft_consensus.cc:2396] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260808 20:14:46.734876 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:46.741518 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:46.742317 12249 raft_consensus.cc:2379] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:46.743000 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:46.743628 12249 raft_consensus.cc:2379] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 5: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:46.737340 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:46.734792 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:46.747342 12243 raft_consensus.cc:2396] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 20:14:46.729152 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:46.747828 12038 raft_consensus.cc:1226] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:46.748118 12245 raft_consensus.cc:2379] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 2: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
W20260808 20:14:46.750073 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 94889.1us
I20260808 20:14:46.751015 13015 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:46.751791 15997 consensus_queue.cc:1035] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 20:14:46.752830 15972 raft_consensus.cc:2781] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:46.684093 13076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53290 (request call id 585) took 1611 ms. Trace:
I20260808 20:14:46.753564 15972 raft_consensus.cc:686] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:46.753921 13076 rpcz_store.cc:276] 0808 20:14:45.072503 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:45.732040 (+659537us) service_pool.cc:227] Handling call
0808 20:14:46.683704 (+951664us) spinlock_profiling.cc:243] Waited 390 ms on lock 0x7b54004bf5d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:46.683894 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":896155904}
I20260808 20:14:46.754731 15972 consensus_queue.cc:237] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:46.755908 16021 raft_consensus.cc:482] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:46.756613 16021 raft_consensus.cc:504] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:46.752283 13015 leader_election.cc:302] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:46.751298 12000 leader_election.cc:302] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:46.764358 12150 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:46.769086 12041 raft_consensus.cc:2396] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260808 20:14:46.768997 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:46.773751 16064 raft_consensus.cc:686] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:46.774248 15949 raft_consensus.cc:2781] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:46.774830 15949 raft_consensus.cc:686] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:46.775233 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:46.719053 12251 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:14:46.772436 12207 leader_election.cc:302] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:46.851331 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:46.782323 13072 raft_consensus.cc:1226] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7. Ops: []
W20260808 20:14:46.880457 11894 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:57488 (request call id 320) took 19204 ms (client timeout 15000 ms). Trace:
I20260808 20:14:46.909740 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:46.787650 15776 consensus_meta.cc:287] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.720s user 0.010s sys 0.002s
W20260808 20:14:46.850323 13074 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53290 (request call id 586) took 1777 ms (client timeout 1757 ms). Trace:
W20260808 20:14:46.852198 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 62569.4us
I20260808 20:14:46.774852 16064 consensus_queue.cc:237] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:46.844235 11897 catalog_manager.cc:5495] T e020c14c3aeb419493cd543071ed7801 P e5bd9a4616ce4acfa20dc7a002984a20 reported cstate change: term changed from 1 to 4, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4). New cstate: current_term: 4 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:14:46.934336 15994 consensus_queue.cc:1035] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 20:14:46.935014 15994 consensus_queue.cc:1225] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer e5bd9a4616ce4acfa20dc7a002984a20 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260808 20:14:46.849623 15886 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:46.938452 13074 rpcz_store.cc:269] 0808 20:14:45.072753 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:45.464003 (+391250us) service_pool.cc:227] Handling call
0808 20:14:46.849955 (+1385952us) spinlock_profiling.cc:243] Waited 642 ms on lock 0x7b54006e1254. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:46.850127 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1477117184}
I20260808 20:14:46.775738 12102 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:46.776142 12243 raft_consensus.cc:2379] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 7: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:46.841401 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:46.936722 12001 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:46.783321 15931 raft_consensus.cc:2781] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260808 20:14:46.947805 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 226488us
W20260808 20:14:46.909337 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 158129us
I20260808 20:14:46.932097 12245 raft_consensus.cc:2354] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 6. Current term is 7.
I20260808 20:14:46.782758 13015 leader_election.cc:302] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:46.929947 12251 raft_consensus.cc:595] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:46.912570 15997 raft_consensus.cc:2781] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:46.783128 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:46.946653 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 322157us
I20260808 20:14:46.947032 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:46.948719 13016 leader_election.cc:334] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.788s (SENT)
I20260808 20:14:46.781479 16021 leader_election.cc:288] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:46.966378 12207 leader_election.cc:302] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:46.942622 11994 leader_election.cc:334] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Timed out: RequestConsensusVote RPC to 127.10.24.3:38391 timed out after 2.009s (SENT)
W20260808 20:14:46.962118 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 186485us
I20260808 20:14:46.963384 12251 raft_consensus.cc:603] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 20:14:46.786393 15908 raft_consensus.cc:482] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
W20260808 20:14:46.992067 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 73694.6us
I20260808 20:14:46.929746 15614 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:46.991895 12249 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:46.948777 15941 raft_consensus.cc:2726] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:46.947750 15931 raft_consensus.cc:482] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:46.932807 15776 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:46.998916 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:47.002372 12249 raft_consensus.cc:2379] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 2: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:46.999086 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:46.992368 11994 leader_election.cc:302] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:46.968623 15997 raft_consensus.cc:482] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:46.999742 13014 leader_election.cc:398] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Vote denied by peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) with higher term. Message: Invalid argument: T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 6. Current term is 7.
I20260808 20:14:46.993747 15908 raft_consensus.cc:3037] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:47.006886 13014 leader_election.cc:401] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260808 20:14:46.999300 15886 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:47.009160 12241 raft_consensus.cc:2354] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5.
W20260808 20:14:47.004122 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 67575.5us
I20260808 20:14:47.001492 15931 raft_consensus.cc:3037] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:47.015997 12000 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:47.014886 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
W20260808 20:14:47.016371 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 95647.6us
I20260808 20:14:47.020902 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:47.022142 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:47.023435 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 57188.1us
I20260808 20:14:47.008020 15997 raft_consensus.cc:3037] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
W20260808 20:14:47.028687 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 139702us
I20260808 20:14:47.029882 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:46.775990 15949 consensus_queue.cc:237] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.014719 15614 raft_consensus.cc:2726] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:47.036120 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:47.041862 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:47.014991 15666 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:47.027051 15923 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Advancing to term 7
W20260808 20:14:47.112005 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 66669us
W20260808 20:14:47.112519 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 109925us
I20260808 20:14:47.006422 15948 consensus_queue.cc:1046] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260808 20:14:47.114779 12251 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
W20260808 20:14:47.115381 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 71626.7us
I20260808 20:14:47.123255 12245 raft_consensus.cc:1226] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260808 20:14:47.112954 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 330251us
W20260808 20:14:47.127449 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 82402.4us
I20260808 20:14:47.133581 13015 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:47.134686 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 91068.2us
I20260808 20:14:47.138115 12046 raft_consensus.cc:1226] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:47.134398 12105 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
W20260808 20:14:46.932085 11894 rpcz_store.cc:269] 0808 20:14:27.675780 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:27.676023 (+ 243us) service_pool.cc:227] Handling call
0808 20:14:27.753932 (+ 77909us) spinlock_profiling.cc:243] Waited 67.4 ms on lock 0x7b5400002dd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825faf0 00007fa9682e2f2a 00007fa96e234d6c 00007fa96e32ec3e 00007fa96c640fb4 00007fa96c640f42 00007fa96c640e71 00007fa96c640dec 00007fa96c6400a2 00007fa96670494c 00007fa966703d86
0808 20:14:46.880161 (+19126229us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 20:14:37.037786 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 20:14:37.112264 (+ 74478us) write_op.cc:432] Acquiring schema lock in shared mode
0808 20:14:37.121140 (+ 8876us) write_op.cc:435] Acquired schema lock
0808 20:14:37.124910 (+ 3770us) tablet.cc:660] Decoding operations
0808 20:14:37.218119 (+ 93209us) write_op.cc:620] Acquiring the partition lock for write op
0808 20:14:37.271604 (+ 53485us) write_op.cc:641] Partition lock acquired for write op
0808 20:14:37.271639 (+ 35us) tablet.cc:683] Acquiring locks for 2 operations
0808 20:14:37.395978 (+124339us) tablet.cc:699] Row locks acquired
0808 20:14:37.407365 (+ 11387us) write_op.cc:260] PREPARE: finished
0808 20:14:37.414691 (+ 7326us) write_op.cc:270] Start()
0808 20:14:37.414950 (+ 259us) write_op.cc:276] Timestamp: P: 1786220077414621 usec, L: 0
0808 20:14:37.414990 (+ 40us) op_driver.cc:331] REPLICATION: starting
0808 20:14:37.415647 (+ 657us) log.cc:842] Serialized 912 byte log entry
0808 20:14:38.265464 (+849817us) op_driver.cc:444] REPLICATION: finished
0808 20:14:39.628299 (+1362835us) spinlock_profiling.cc:243] Waited 3.29 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa9653945fe 00007fa96e1cba39 00007fa9653b18bb 00007fa9653afcbf 00007fa9653b1e22 00007fa96bcea410
0808 20:14:40.575845 (+947546us) write_op.cc:301] APPLY: starting
0808 20:14:40.576037 (+ 192us) tablet.cc:1365] starting BulkCheckPresence
0808 20:14:40.576267 (+ 230us) tablet.cc:1368] finished BulkCheckPresence
0808 20:14:40.576292 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 20:14:40.792450 (+216158us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 20:14:40.825708 (+ 33258us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0808 20:14:40.825756 (+ 48us) write_op.cc:312] APPLY: finished
0808 20:14:40.826056 (+ 300us) log.cc:842] Serialized 29 byte log entry
0808 20:14:41.886213 (+1060157us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:14:41.898705 (+ 12492us) write_op.cc:454] Released schema lock
0808 20:14:41.948712 (+ 50007us) write_op.cc:341] FINISH: Updating metrics
0808 20:14:41.954008 (+ 5296us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:14:41.954038 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3728,"reported_tablets":22,"spinlock_wait_cycles":154913152,"child_traces":[["op",{"apply.queue_time_us":2275488,"apply.run_cpu_time_us":44800,"apply.run_wall_time_us":1378500,"num_ops":2,"prepare.queue_time_us":204066,"prepare.run_cpu_time_us":27080,"prepare.run_wall_time_us":473403,"raft.queue_time_us":311719,"raft.run_cpu_time_us":50050,"raft.run_wall_time_us":3541292,"replication_time_us":824079,"spinlock_wait_cycles":7571840,"thread_start_us":4743371,"threads_started":5,"wal-append.queue_time_us":886750,"wal-append.run_cpu_time_us":38305,"wal-append.run_wall_time_us":3093893}]]}
I20260808 20:14:47.124593 15690 raft_consensus.cc:3037] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:47.145210 12207 leader_election.cc:302] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:47.140502 12101 leader_election.cc:302] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:47.158016 12041 raft_consensus.cc:1261] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260808 20:14:47.159405 11994 leader_election.cc:334] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Timed out: RequestConsensusVote RPC to 127.10.24.3:38391 timed out after 2.049s (SENT)
I20260808 20:14:47.161664 15987 consensus_queue.cc:1035] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.118s
I20260808 20:14:47.162201 15699 consensus_queue.cc:1046] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.136s
I20260808 20:14:47.163906 15667 raft_consensus.cc:3032] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:47.164247 15667 raft_consensus.cc:729] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:47.173336 11994 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:47.136662 15951 raft_consensus.cc:2781] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:14:47.177418 13015 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:47.161609 15107 raft_consensus.cc:2781] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:47.182777 15107 raft_consensus.cc:686] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:47.183501 15906 raft_consensus.cc:2726] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:47.183857 15107 consensus_queue.cc:237] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.186740 12138 raft_consensus.cc:1226] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:47.175590 15968 raft_consensus.cc:2726] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 20:14:47.191049 15592 raft_consensus.cc:2726] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:47.192090 15835 raft_consensus.cc:3032] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:47.192519 15835 raft_consensus.cc:729] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:47.193671 15835 consensus_queue.cc:260] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:47.192217 16078 log.cc:924] Time spent T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260808 20:14:47.198048 15906 consensus_queue.cc:1046] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 20:14:47.198321 15835 raft_consensus.cc:3037] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:47.198712 15614 raft_consensus.cc:2726] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 20:14:47.200624 16054 log.cc:924] Time spent T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.472s user 0.003s sys 0.000s
I20260808 20:14:47.202318 16005 raft_consensus.cc:2781] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:47.203284 15971 raft_consensus.cc:2726] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:47.198042 15776 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:14:47.204262 16074 log.cc:924] Time spent T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260808 20:14:47.159570 16014 raft_consensus.cc:2726] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:47.147289 15873 consensus_queue.cc:1035] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260808 20:14:47.181380 15951 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:47.207516 12247 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 453) took 1364 ms. Trace:
I20260808 20:14:47.249856 12247 rpcz_store.cc:276] 0808 20:14:45.843032 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:45.897902 (+ 54870us) service_pool.cc:227] Handling call
0808 20:14:46.075584 (+177682us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:46.076217 (+ 633us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:47.207147 (+1130930us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b5400375350. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d9a09 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:47.207350 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":378396032}
I20260808 20:14:47.165196 15667 consensus_queue.cc:260] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.251660 15667 raft_consensus.cc:3037] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:47.189117 12138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1699) took 1364 ms. Trace:
I20260808 20:14:47.253108 12138 rpcz_store.cc:276] 0808 20:14:45.824669 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:45.834940 (+ 10271us) service_pool.cc:227] Handling call
0808 20:14:47.186087 (+1351147us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:47.186386 (+ 299us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:47.187201 (+ 815us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:47.189028 (+ 1827us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24451840}
I20260808 20:14:47.255436 13072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 492) took 1105 ms. Trace:
I20260808 20:14:47.255844 13072 rpcz_store.cc:276] 0808 20:14:46.149608 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:46.169134 (+ 19526us) service_pool.cc:227] Handling call
0808 20:14:46.714639 (+545505us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:46.931992 (+217353us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:47.255195 (+323203us) spinlock_profiling.cc:243] Waited 516 ms on lock 0x7b54004bf5d0. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d9a09 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:47.255363 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1185665792}
W20260808 20:14:47.194070 16068 log.cc:924] Time spent T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.147s user 0.001s sys 0.000s
W20260808 20:14:47.213673 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 169012us
I20260808 20:14:47.259543 12105 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:47.260869 12105 leader_election.cc:302] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:47.269104 15948 raft_consensus.cc:2781] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:47.269565 15948 raft_consensus.cc:686] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:47.270493 15948 consensus_queue.cc:237] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.272048 16071 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:47.223505 15602 raft_consensus.cc:2781] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 20:14:47.272939 15602 raft_consensus.cc:686] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:47.273886 15602 consensus_queue.cc:237] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.221081 12250 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8.
W20260808 20:14:47.223359 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 46010us
I20260808 20:14:47.203470 15908 raft_consensus.cc:504] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.272603 16071 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.263821 15826 raft_consensus.cc:2726] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 20:14:47.264437 15653 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:47.264997 15934 consensus_queue.cc:1046] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20260808 20:14:47.275884 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:47.281958 12205 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:47.137106 15666 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.314059 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:47.297434 15951 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:47.334611 12046 raft_consensus.cc:1226] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:47.335011 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
W20260808 20:14:47.336418 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 134933us
W20260808 20:14:47.337947 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 213432us
I20260808 20:14:47.343567 12150 raft_consensus.cc:2396] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
W20260808 20:14:47.344399 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 206679us
I20260808 20:14:47.305795 16005 raft_consensus.cc:686] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:47.346844 12254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 452) took 1294 ms. Trace:
I20260808 20:14:47.349723 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:47.359862 12101 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:47.361377 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 248681us
I20260808 20:14:47.361348 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:47.343734 15987 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:47.389998 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:47.388077 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:47.393190 12041 raft_consensus.cc:1261] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 20:14:47.392755 12039 raft_consensus.cc:2354] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 6. Current term is 7.
I20260808 20:14:47.396852 15886 raft_consensus.cc:2726] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:47.398324 12000 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:47.400041 15699 raft_consensus.cc:3032] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:47.400437 15699 raft_consensus.cc:729] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:47.401235 12138 raft_consensus.cc:2396] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260808 20:14:47.401878 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:47.402249 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:47.403134 12140 raft_consensus.cc:2379] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
W20260808 20:14:47.403560 16083 log.cc:924] Time spent T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260808 20:14:47.404119 12250 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:14:47.409308 15771 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:47.409970 15771 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.353701 12254 rpcz_store.cc:276] 0808 20:14:45.842791 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:45.886127 (+ 43336us) service_pool.cc:227] Handling call
0808 20:14:45.896312 (+ 10185us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:45.896540 (+ 228us) raft_consensus.cc:1520] Early marking committed up to index 4
0808 20:14:45.914938 (+ 18398us) log.cc:842] Serialized 14 byte log entry
0808 20:14:46.791838 (+876900us) spinlock_profiling.cc:243] Waited 4.75 ms on lock 0x7b1800463060. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96e2518b1 00007fa96e25e661 00007fa965394345 00007fa96e1cba39 00007fa9653b18bb 00007fa9653afcbf 00007fa9653b1e22 00007fa967f5b5bb 00007fa967f61063 00007fa967f6157b 00007fa9682e488a
0808 20:14:47.035115 (+243277us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:47.109621 (+ 74506us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:47.135183 (+ 25562us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:47.135257 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:47.137260 (+ 2003us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":11641,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":12093,"raft.queue_time_us":6774,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":4851,"spinlock_wait_cycles":10913536,"thread_start_us":222820,"threads_started":1}
I20260808 20:14:47.410542 16005 consensus_queue.cc:237] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.401324 15699 consensus_queue.cc:260] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.415169 15699 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:47.417852 15666 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:47.419536 16071 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:47.420586 12144 raft_consensus.cc:1226] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 2. Current term is 3. Ops: []
I20260808 20:14:47.421850 15142 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:47.353765 15776 raft_consensus.cc:2726] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260808 20:14:47.387595 13016 leader_election.cc:334] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.771s (SENT)
I20260808 20:14:47.358695 13074 pending_rounds.cc:76] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20: Aborting all ops after (but not including) 6
I20260808 20:14:47.455670 12244 raft_consensus.cc:2379] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 8: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
W20260808 20:14:47.410359 16068 log.cc:924] Time spent T b7e4c0cacf074dee848eed1ab4d27ebf P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.153s user 0.004s sys 0.002s
I20260808 20:14:47.446697 12046 raft_consensus.cc:1261] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260808 20:14:47.353102 16014 raft_consensus.cc:482] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:47.454033 12250 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 9.
W20260808 20:14:47.457069 11994 leader_election.cc:334] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Timed out: RequestConsensusVote RPC to 127.10.24.3:38391 timed out after 2.155s (SENT)
I20260808 20:14:47.457484 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:47.460577 12150 raft_consensus.cc:2362] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 9. Re-sending same reply.
I20260808 20:14:47.461380 11994 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:47.458806 12046 pending_rounds.cc:76] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting all ops after (but not including) 4
I20260808 20:14:47.464632 12046 pending_rounds.cc:97] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260808 20:14:47.464882 12046 raft_consensus.cc:2866] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:47.462771 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:47.454778 13074 pending_rounds.cc:97] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260808 20:14:47.469376 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:47.471004 15923 raft_consensus.cc:2726] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) with higher term. Message: Invalid argument: T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 6. Current term is 7.
I20260808 20:14:47.449258 15987 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.472860 15771 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:47.466728 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:47.473125 12036 raft_consensus.cc:1226] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 8. Current term is 9. Ops: []
I20260808 20:14:47.424710 15931 raft_consensus.cc:504] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.478803 15941 consensus_queue.cc:1035] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:14:47.479727 12041 raft_consensus.cc:1226] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 5. Current term is 6. Ops: []
W20260808 20:14:47.482421 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 131832us
I20260808 20:14:47.484094 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:47.484967 15934 raft_consensus.cc:482] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:47.490419 12250 raft_consensus.cc:1261] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 20:14:47.490989 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:47.492307 16014 raft_consensus.cc:504] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:47.493908 15624 raft_consensus.cc:2726] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:47.495855 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:47.496135 15951 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:47.496544 15994 raft_consensus.cc:2726] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 20:14:47.466296 15699 raft_consensus.cc:2726] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:47.498325 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:47.493438 12141 raft_consensus.cc:2362] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5. Re-sending same reply.
W20260808 20:14:47.469367 16089 log.cc:924] Time spent T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.192s user 0.002s sys 0.002s
I20260808 20:14:47.510396 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:47.513753 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 160710us
I20260808 20:14:47.493964 15892 raft_consensus.cc:3032] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:47.501456 13074 raft_consensus.cc:2866] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 20:14:47.518294 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 123728us
I20260808 20:14:47.519060 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:47.501355 15592 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:47.538280 12144 raft_consensus.cc:2362] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6. Re-sending same reply.
I20260808 20:14:47.524905 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:47.530206 12146 raft_consensus.cc:2454] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:47.527859 12040 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:14:47.531620 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:47.532224 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:47.539067 12101 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:47.539464 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:47.501510 15971 consensus_queue.cc:1046] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20260808 20:14:47.517066 16049 consensus_queue.cc:1035] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260808 20:14:47.527730 15908 leader_election.cc:288] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:47.544232 12102 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:47.521939 13076 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8.
I20260808 20:14:47.524873 15997 raft_consensus.cc:504] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.551829 12038 raft_consensus.cc:1226] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:47.549840 12244 raft_consensus.cc:3037] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:47.552243 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:47.552091 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:47.537135 15892 raft_consensus.cc:729] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:47.550928 12147 raft_consensus.cc:2362] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6. Re-sending same reply.
I20260808 20:14:47.552815 12146 raft_consensus.cc:2362] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5. Re-sending same reply.
I20260808 20:14:47.538069 15666 consensus_queue.cc:1035] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
W20260808 20:14:47.594878 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 55904.4us
I20260808 20:14:47.573714 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:47.595712 12101 leader_election.cc:302] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:47.596467 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
W20260808 20:14:47.597225 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 48392.6us
I20260808 20:14:47.597656 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:47.598250 12150 raft_consensus.cc:2362] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6. Re-sending same reply.
I20260808 20:14:47.599041 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:47.599670 12150 raft_consensus.cc:2354] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for earlier term 7. Current term is 8.
I20260808 20:14:47.599992 11994 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:47.600383 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:47.601151 15931 leader_election.cc:288] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:47.601361 16008 consensus_queue.cc:1035] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 20:14:47.602710 16008 raft_consensus.cc:2726] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:47.571360 15592 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:47.604389 11994 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:47.530166 15906 consensus_queue.cc:1046] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.022s
I20260808 20:14:47.595980 12141 raft_consensus.cc:2396] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260808 20:14:47.597142 12138 raft_consensus.cc:1226] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260808 20:14:47.639650 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:47.641232 12137 raft_consensus.cc:2454] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8.
I20260808 20:14:47.620565 12001 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:47.623020 15592 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:47.623422 12101 leader_election.cc:302] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:47.623698 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:47.624339 15743 raft_consensus.cc:2741] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260808 20:14:47.625152 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:47.646051 12146 raft_consensus.cc:2396] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 20:14:47.625277 13016 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:47.528363 15934 raft_consensus.cc:504] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.625725 16014 leader_election.cc:288] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:47.648643 15653 raft_consensus.cc:2726] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:47.626149 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:47.649840 12147 raft_consensus.cc:2379] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 6: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:47.649978 15934 leader_election.cc:288] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:47.650262 12101 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:47.627075 15892 consensus_queue.cc:260] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.651141 15892 raft_consensus.cc:3037] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:47.651474 15762 raft_consensus.cc:2726] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 20:14:47.652508 16090 raft_consensus.cc:3037] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:47.651052 15699 raft_consensus.cc:2726] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:47.654309 12101 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:47.645078 15873 raft_consensus.cc:2781] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 20:14:47.655148 15699 raft_consensus.cc:2781] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:47.655421 15873 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:47.655604 15699 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:47.655851 15873 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:14:47.656026 15699 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:47.657868 15987 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:47.650985 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:47.627360 15971 consensus_queue.cc:1046] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 20:14:47.628684 15951 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:47.628839 16021 raft_consensus.cc:3032] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:47.603106 15997 leader_election.cc:288] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:47.613411 15826 raft_consensus.cc:2781] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:47.677410 15826 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:47.635921 15142 raft_consensus.cc:482] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:47.625775 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:47.604292 15776 raft_consensus.cc:482] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:47.617369 15674 raft_consensus.cc:2726] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:47.673353 16021 raft_consensus.cc:729] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:47.679965 12144 raft_consensus.cc:2379] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
W20260808 20:14:47.689110 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 84759.7us
W20260808 20:14:47.690603 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 72652.3us
I20260808 20:14:47.677795 15826 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:47.680089 15987 raft_consensus.cc:3032] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Stepping down as leader of term 6
I20260808 20:14:47.681407 15614 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:47.711001 12101 leader_election.cc:302] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:47.714156 11994 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:47.683019 15142 raft_consensus.cc:504] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:47.689980 15776 raft_consensus.cc:504] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:47.702747 15987 raft_consensus.cc:729] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:47.737648 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:47.738683 16061 raft_consensus.cc:2781] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:47.739260 16061 raft_consensus.cc:686] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:47.740163 16061 consensus_queue.cc:237] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.741865 15699 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:47.743904 15674 raft_consensus.cc:3032] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 20:14:47.744117 15699 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:47.744323 15674 raft_consensus.cc:729] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:47.713974 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:47.717206 15997 raft_consensus.cc:2726] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:47.757946 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:47.755585 12147 raft_consensus.cc:1226] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260808 20:14:47.755728 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:47.758623 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 251546us
I20260808 20:14:47.759250 12038 raft_consensus.cc:2396] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20260808 20:14:47.757493 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 46758.4us
I20260808 20:14:47.761624 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:47.710353 15614 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.762372 12146 raft_consensus.cc:2379] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 7: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
W20260808 20:14:47.747792 15690 consensus_meta.cc:287] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.569s user 0.010s sys 0.000s
W20260808 20:14:47.769814 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 68827.6us
I20260808 20:14:47.775221 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:47.777570 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:47.780279 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:47.787969 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 389739us
I20260808 20:14:47.761343 15931 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:47.792740 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:47.773068 15690 raft_consensus.cc:2726] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:47.801123 12247 raft_consensus.cc:1226] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:47.747822 16021 consensus_queue.cc:260] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:47.811359 11994 leader_election.cc:302] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:47.754913 15674 consensus_queue.cc:260] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:47.813800 16008 raft_consensus.cc:2726] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:47.814272 11994 leader_election.cc:302] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:47.815908 11994 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:47.817153 16094 raft_consensus.cc:2726] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:47.805310 15987 consensus_queue.cc:260] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.817461 12000 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:47.817806 16094 raft_consensus.cc:2726] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:47.819609 16090 raft_consensus.cc:2726] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:47.820362 15776 leader_election.cc:288] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:47.800307 15931 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:47.801146 15835 consensus_meta.cc:287] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.575s user 0.005s sys 0.012s
I20260808 20:14:47.836836 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:47.837870 13071 raft_consensus.cc:2396] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260808 20:14:47.838295 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:47.838874 13015 leader_election.cc:398] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Vote denied by peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747) with higher term. Message: Invalid argument: T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 6. Current term is 7.
I20260808 20:14:47.842291 16021 raft_consensus.cc:3037] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:47.845513 16005 consensus_queue.cc:1046] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 20:14:47.846448 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:47.846699 13076 raft_consensus.cc:1226] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 LEADER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:47.847047 16005 raft_consensus.cc:3032] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:47.847561 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:47.849116 12138 raft_consensus.cc:1261] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 20:14:47.847597 16005 raft_consensus.cc:729] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:47.852622 15142 leader_election.cc:288] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:47.850481 12038 raft_consensus.cc:1261] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 20:14:47.853201 16005 consensus_queue.cc:260] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.854691 16005 raft_consensus.cc:3037] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:47.855778 12146 raft_consensus.cc:1226] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:47.840503 15646 consensus_queue.cc:1035] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 20:14:47.857563 12141 raft_consensus.cc:1261] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 20:14:47.830332 15931 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:47.830956 13079 raft_consensus.cc:1261] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 20:14:47.831688 15674 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:14:47.832199 16008 raft_consensus.cc:2781] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 20:14:47.833292 13016 leader_election.cc:302] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:47.830374 12000 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:47.833379 13072 raft_consensus.cc:1226] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:47.834872 12244 raft_consensus.cc:2454] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:47.835886 13075 raft_consensus.cc:1226] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:47.850841 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:47.850154 12206 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:47.874368 12040 raft_consensus.cc:1226] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:47.873054 12047 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6.
I20260808 20:14:47.879962 13015 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:47.880302 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:47.886376 12254 raft_consensus.cc:1226] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:47.851992 15614 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:47.888415 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:47.893965 13079 raft_consensus.cc:2454] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:47.894418 13014 leader_election.cc:302] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:47.893262 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:47.897919 12001 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:47.900516 12247 raft_consensus.cc:2454] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6.
I20260808 20:14:47.901293 13014 leader_election.cc:302] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
W20260808 20:14:47.904675 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 71483.3us
I20260808 20:14:47.905175 15951 consensus_queue.cc:1046] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:47.905557 13016 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:47.905754 15614 raft_consensus.cc:2781] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260808 20:14:47.906159 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 56123.8us
I20260808 20:14:47.906309 15951 raft_consensus.cc:3032] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:47.907004 15951 raft_consensus.cc:729] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:47.907872 15951 consensus_queue.cc:260] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:47.909036 15951 raft_consensus.cc:3037] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:47.893236 15142 raft_consensus.cc:2726] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 20:14:47.912106 15949 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:47.906244 15614 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:47.912679 15949 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.913349 15614 raft_consensus.cc:3037] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:47.883025 16008 raft_consensus.cc:686] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:47.919096 16008 consensus_queue.cc:237] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:47.921530 15949 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:47.923647 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:47.883359 16061 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:47.925529 12038 raft_consensus.cc:1261] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 20:14:47.925251 16061 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.927304 16061 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:47.927853 16061 consensus_queue.cc:1035] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 20:14:47.929028 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:47.870637 15892 consensus_queue.cc:1035] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260808 20:14:47.931875 15826 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.932364 15614 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:47.888922 15931 raft_consensus.cc:2781] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:47.933450 15931 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:47.933789 15931 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:47.896082 15690 raft_consensus.cc:2781] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:47.934870 15690 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:47.935199 15690 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
W20260808 20:14:47.907232 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 76937.2us
I20260808 20:14:47.906898 12206 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:47.937171 15951 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
W20260808 20:14:47.946732 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 44244.4us
I20260808 20:14:47.945180 12000 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:47.901897 15107 raft_consensus.cc:2726] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:47.951359 11994 leader_election.cc:302] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:47.954008 13014 leader_election.cc:302] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:47.900874 16108 raft_consensus.cc:2726] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:14:47.950852 15653 raft_consensus.cc:2781] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:47.969652 12244 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:14:47.970276 12244 raft_consensus.cc:595] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:47.970649 12244 raft_consensus.cc:603] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 20:14:47.971441 12247 raft_consensus.cc:1226] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260808 20:14:47.955570 15886 raft_consensus.cc:2726] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:47.956578 15908 raft_consensus.cc:2781] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:47.973692 12036 raft_consensus.cc:1226] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:47.954831 15142 raft_consensus.cc:2726] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 20:14:47.978159 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:47.975766 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 70284us
I20260808 20:14:47.956456 15892 raft_consensus.cc:2726] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:47.981858 11994 leader_election.cc:302] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:47.976876 16021 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:47.977339 15653 raft_consensus.cc:686] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:47.977269 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:47.980110 12243 raft_consensus.cc:1226] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260808 20:14:47.977205 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:47.946337 15951 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:47.982620 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:47.991921 16093 raft_consensus.cc:2726] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:47.993028 16098 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:47.993498 16098 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:47.995585 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:47.979714 15908 raft_consensus.cc:482] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:14:48.002947 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 96164.5us
I20260808 20:14:48.008708 13016 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:48.017179 12036 raft_consensus.cc:2454] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:47.983115 15987 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:47.965564 15873 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.025158 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:48.027930 12047 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:48.025784 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:48.028635 12254 raft_consensus.cc:2379] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 7: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:48.029796 12206 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:47.995409 16098 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:47.997087 16090 raft_consensus.cc:2726] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:48.032523 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:48.044368 12048 raft_consensus.cc:2379] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 9: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:48.033632 12243 raft_consensus.cc:1226] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6. Ops: []
W20260808 20:14:48.044227 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 62446us
I20260808 20:14:48.046105 11994 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:48.014907 15908 raft_consensus.cc:3037] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:48.048780 12036 raft_consensus.cc:1226] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:48.050729 12205 leader_election.cc:302] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
W20260808 20:14:48.057912 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 49295.1us
I20260808 20:14:48.058749 12254 raft_consensus.cc:1226] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6. Ops: []
W20260808 20:14:48.060566 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 84821.5us
I20260808 20:14:48.004077 16021 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:48.042939 15873 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:47.995360 15653 consensus_queue.cc:237] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.072363 11994 leader_election.cc:302] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:48.048410 15892 raft_consensus.cc:2781] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:48.073629 12206 leader_election.cc:302] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:48.093883 12036 raft_consensus.cc:1226] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:48.054368 15826 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:48.069465 16090 raft_consensus.cc:2726] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 20:14:48.069589 15971 raft_consensus.cc:2781] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:48.101630 12105 leader_election.cc:302] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:48.098186 12000 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:48.048271 15592 consensus_queue.cc:1046] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260808 20:14:48.115916 12244 raft_consensus.cc:1261] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 20:14:48.032965 16134 consensus_queue.cc:1046] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.080s
I20260808 20:14:48.130615 16090 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:48.131124 12000 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:48.131873 16098 raft_consensus.cc:2726] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:48.132340 12000 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:48.133100 16098 raft_consensus.cc:2726] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:48.133533 12000 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:48.134744 12000 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:48.139780 12150 raft_consensus.cc:1261] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 20:14:48.136504 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:48.140512 12146 raft_consensus.cc:2379] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 7: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:48.137117 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:48.095909 15614 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:48.089337 15892 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:48.135682 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:48.093739 15997 raft_consensus.cc:2726] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:48.138891 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:48.094956 15107 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:48.144805 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:48.145853 12248 raft_consensus.cc:1226] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 6. Current term is 7. Ops: []
I20260808 20:14:48.141881 12141 raft_consensus.cc:2454] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6.
I20260808 20:14:48.071781 15646 consensus_queue.cc:1046] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.067s
I20260808 20:14:48.150277 12244 raft_consensus.cc:1226] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:48.129185 15971 raft_consensus.cc:482] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:48.097879 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:48.104947 16061 consensus_queue.cc:1046] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.045s
I20260808 20:14:48.146622 16118 raft_consensus.cc:2781] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election won for term 6
W20260808 20:14:48.170125 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 108995us
I20260808 20:14:48.147045 15892 raft_consensus.cc:3037] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:48.146544 15826 raft_consensus.cc:2726] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:48.149310 15614 raft_consensus.cc:2781] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:14:48.171844 12000 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:48.186652 15858 consensus_queue.cc:1046] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 20:14:48.187084 12101 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:48.186642 16064 consensus_queue.cc:1046] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:14:48.187911 15948 raft_consensus.cc:3032] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:14:48.188042 15973 raft_consensus.cc:2726] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 20:14:48.188315 15948 raft_consensus.cc:729] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:48.189317 16064 raft_consensus.cc:3032] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:48.189181 15948 consensus_queue.cc:260] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:48.189684 16064 raft_consensus.cc:729] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:48.190553 15948 raft_consensus.cc:3037] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:48.190429 16064 consensus_queue.cc:260] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:48.191610 16064 raft_consensus.cc:3037] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:48.193229 12101 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:48.193701 12137 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:48.194667 16061 raft_consensus.cc:2726] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 20:14:48.195730 16049 consensus_queue.cc:1046] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:48.162878 15666 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:48.166533 15971 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:48.170488 13074 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:14:48.201745 12248 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:14:48.196808 16061 raft_consensus.cc:3032] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:48.214135 12248 raft_consensus.cc:595] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:48.214576 12244 raft_consensus.cc:1226] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 1. Current term is 2. Ops: []
W20260808 20:14:48.217695 16136 log.cc:924] Time spent T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.164s user 0.000s sys 0.001s
I20260808 20:14:48.202112 16102 raft_consensus.cc:2781] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader election won for term 9
I20260808 20:14:48.219658 12248 raft_consensus.cc:603] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 20:14:48.220292 16102 raft_consensus.cc:686] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 9 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:48.221345 16102 consensus_queue.cc:237] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.10, Last appended by leader: 10, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.169718 15776 raft_consensus.cc:482] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:48.197216 15614 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:48.172814 15951 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:48.157397 15107 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.227221 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:48.201538 16118 raft_consensus.cc:686] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:48.218571 15886 raft_consensus.cc:2741] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260808 20:14:48.229018 15776 raft_consensus.cc:504] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:48.272945 15776 leader_election.cc:288] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:48.273514 15776 consensus_queue.cc:1046] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260808 20:14:48.237581 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 70516.9us
I20260808 20:14:48.274507 15776 consensus_queue.cc:1046] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 20:14:48.275194 12000 leader_election.cc:302] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:48.275358 13074 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 9.
I20260808 20:14:48.277144 15997 raft_consensus.cc:2781] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:48.277683 15997 raft_consensus.cc:686] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:48.278525 15997 consensus_queue.cc:237] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:48.279906 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:48.280529 13075 raft_consensus.cc:3037] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:48.182684 16108 consensus_queue.cc:1046] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20260808 20:14:48.227097 15614 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:48.238286 15951 consensus_queue.cc:1046] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 20:14:48.232237 15602 consensus_queue.cc:1035] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
W20260808 20:14:48.239539 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 68726.3us
I20260808 20:14:48.243474 16090 raft_consensus.cc:2726] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:48.243947 15690 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.245883 15107 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:48.219058 16061 raft_consensus.cc:729] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:48.203486 15666 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.251027 16118 consensus_queue.cc:237] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.257652 15931 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.263600 15674 raft_consensus.cc:3032] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:48.264518 16014 raft_consensus.cc:3032] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:48.232770 15699 consensus_queue.cc:1046] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260808 20:14:48.268682 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:48.269611 15592 consensus_queue.cc:1035] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:48.270397 12039 raft_consensus.cc:1261] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 20:14:48.234624 16021 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:48.256799 12047 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 7.
W20260808 20:14:48.324672 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 91179.1us
I20260808 20:14:48.324991 16173 raft_consensus.cc:3032] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:48.327980 16176 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:48.330747 12248 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:14:48.331426 12043 raft_consensus.cc:1226] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 6. Current term is 7. Ops: []
W20260808 20:14:48.332078 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 188852us
I20260808 20:14:48.275203 15776 consensus_queue.cc:1225] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer e5bd9a4616ce4acfa20dc7a002984a20 log is divergent from this leader: its last log entry 5.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
W20260808 20:14:48.315328 16087 log.cc:924] Time spent T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.050s user 0.003s sys 0.004s
W20260808 20:14:48.337081 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 60273us
I20260808 20:14:48.339290 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:48.344098 12000 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:48.344522 12036 raft_consensus.cc:2454] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6.
I20260808 20:14:48.339735 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:48.340684 12248 raft_consensus.cc:595] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:48.340260 16173 raft_consensus.cc:729] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
W20260808 20:14:48.348191 16141 log.cc:924] Time spent T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045: Append to log took a long time: real 0.179s user 0.004s sys 0.005s
I20260808 20:14:48.349238 12206 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:48.349848 15743 consensus_queue.cc:1046] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:14:48.350950 15743 raft_consensus.cc:3032] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 20:14:48.351272 15743 raft_consensus.cc:729] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:48.346482 16176 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.352049 15743 consensus_queue.cc:260] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:48.353149 16176 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:48.353588 13076 raft_consensus.cc:1261] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260808 20:14:48.353953 16160 log.cc:924] Time spent T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.090s user 0.000s sys 0.006s
I20260808 20:14:48.354557 12036 raft_consensus.cc:1261] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 20:14:48.333566 15674 raft_consensus.cc:729] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:48.351651 12206 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:48.347218 12249 raft_consensus.cc:1261] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 20:14:48.280862 15892 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.356292 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:48.347554 12251 raft_consensus.cc:2454] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:48.359164 13015 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:48.366183 12000 leader_election.cc:302] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:48.359368 12001 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:48.366559 12101 leader_election.cc:302] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:48.333102 16014 raft_consensus.cc:729] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:48.360601 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:48.348032 16162 log.cc:924] Time spent T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.076s user 0.000s sys 0.005s
I20260808 20:14:48.363926 12036 raft_consensus.cc:1226] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 6. Current term is 7. Ops: []
I20260808 20:14:48.338251 16098 raft_consensus.cc:2726] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:48.368116 16021 consensus_queue.cc:1035] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 20:14:48.368685 15690 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:48.369009 16008 consensus_queue.cc:1046] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 20:14:48.369619 16093 raft_consensus.cc:2781] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 20:14:48.370903 12206 leader_election.cc:398] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Vote denied by peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139) with higher term. Message: Invalid argument: T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for earlier term 7. Current term is 8.
I20260808 20:14:48.347543 12248 raft_consensus.cc:603] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 20:14:48.346174 15951 consensus_queue.cc:1035] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.043s
I20260808 20:14:48.410554 12244 raft_consensus.cc:1261] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260808 20:14:48.379163 12254 raft_consensus.cc:2454] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:48.341385 15908 raft_consensus.cc:504] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.343395 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:48.344597 15107 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:48.395246 12205 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:48.369314 15949 raft_consensus.cc:2781] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:48.363914 15776 raft_consensus.cc:2781] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:48.409832 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:48.346701 16090 raft_consensus.cc:2726] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 20:14:48.382809 16176 raft_consensus.cc:2781] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260808 20:14:48.411576 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 74990.9us
I20260808 20:14:48.423358 12102 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:48.349277 15971 raft_consensus.cc:504] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:48.380220 15699 consensus_queue.cc:1035] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260808 20:14:48.409318 15835 raft_consensus.cc:3037] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:48.431684 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:48.411832 13071 raft_consensus.cc:1261] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260808 20:14:48.435562 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 68112.4us
I20260808 20:14:48.438925 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:48.437129 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 71023.5us
I20260808 20:14:48.443320 12000 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:48.413311 15949 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:48.429590 15743 raft_consensus.cc:3037] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 20:14:48.413960 16175 raft_consensus.cc:2726] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 20:14:48.452407 12243 raft_consensus.cc:1226] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:48.420266 16093 raft_consensus.cc:686] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:48.453969 12048 raft_consensus.cc:1261] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260808 20:14:48.426183 16176 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:48.438844 13074 raft_consensus.cc:2379] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 7: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:48.455642 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:48.419349 16173 consensus_queue.cc:260] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.469494 15762 raft_consensus.cc:3032] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 20:14:48.469890 15762 raft_consensus.cc:729] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:48.470643 15762 consensus_queue.cc:260] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:48.471606 15762 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:48.474454 15908 leader_election.cc:288] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:48.475046 16049 raft_consensus.cc:2781] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 20:14:48.476531 15971 leader_election.cc:288] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:48.478485 13077 raft_consensus.cc:1261] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260808 20:14:48.460579 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 89807.4us
I20260808 20:14:48.379675 16061 consensus_queue.cc:260] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:48.429281 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:48.465690 13078 raft_consensus.cc:1226] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 5. Current term is 6. Ops: [3.5-5.6]
I20260808 20:14:48.419641 15107 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.466848 16093 consensus_queue.cc:237] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.468232 16176 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:48.469521 12036 raft_consensus.cc:2379] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 7: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:48.482379 12206 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:48.453629 15949 raft_consensus.cc:3037] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:48.532860 13080 raft_consensus.cc:1261] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260808 20:14:48.514882 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 69572.3us
I20260808 20:14:48.518447 12105 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:48.523524 16146 log.cc:924] Time spent T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.354s user 0.002s sys 0.000s
I20260808 20:14:48.522075 13016 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:48.540815 12036 raft_consensus.cc:1226] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:48.524677 13077 raft_consensus.cc:1261] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 20:14:48.490360 16049 raft_consensus.cc:686] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:48.524977 15107 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:48.525280 15602 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:48.520821 15987 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:14:48.541033 13016 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:48.413234 15674 consensus_queue.cc:260] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.464675 16008 raft_consensus.cc:2781] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader election won for term 8
W20260808 20:14:48.556206 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 161027us
I20260808 20:14:48.532406 15666 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:48.565100 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 82806.5us
I20260808 20:14:48.546952 15776 raft_consensus.cc:686] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:48.590010 12206 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:48.590770 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 203357us
I20260808 20:14:48.550280 15931 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:48.611510 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:48.552757 15699 raft_consensus.cc:2726] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:48.482571 15934 consensus_queue.cc:1035] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.077s
I20260808 20:14:48.622864 12036 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:48.564460 15592 raft_consensus.cc:2726] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 20:14:48.451683 16014 consensus_queue.cc:260] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.628885 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:48.631249 12251 raft_consensus.cc:2454] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:48.631574 12206 leader_election.cc:302] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:48.634843 12245 raft_consensus.cc:1226] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 6. Current term is 7. Ops: []
I20260808 20:14:48.611120 16008 raft_consensus.cc:686] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:48.546541 16169 consensus_queue.cc:1035] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.083s
I20260808 20:14:48.637081 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:48.634361 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:48.620795 15892 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:48.557065 15602 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.560748 15951 consensus_queue.cc:1035] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.069s
I20260808 20:14:48.634171 15666 raft_consensus.cc:2781] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:48.590003 16194 consensus_queue.cc:1046] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.085s
I20260808 20:14:48.645486 16163 raft_consensus.cc:2726] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260808 20:14:48.635819 16184 log.cc:924] Time spent T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.101s user 0.000s sys 0.001s
I20260808 20:14:48.656565 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:48.637020 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260808 20:14:48.658311 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 234057us
I20260808 20:14:48.669960 15891 consensus_queue.cc:1035] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260808 20:14:48.679030 16159 log.cc:924] Time spent T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.452s user 0.002s sys 0.000s
I20260808 20:14:48.683415 16178 consensus_queue.cc:1046] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 20:14:48.686009 13072 raft_consensus.cc:1226] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:48.689859 16090 raft_consensus.cc:3032] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 20:14:48.697387 13014 leader_election.cc:302] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
W20260808 20:14:48.692602 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 151635us
I20260808 20:14:48.693696 15142 consensus_queue.cc:1046] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
W20260808 20:14:48.697472 16185 log.cc:924] Time spent T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
I20260808 20:14:48.697728 15762 consensus_queue.cc:1046] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
W20260808 20:14:48.698267 16181 log.cc:924] Time spent T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260808 20:14:48.699354 15602 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:48.696835 16200 raft_consensus.cc:2726] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 20:14:48.723503 12105 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:48.724742 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:48.705821 15948 consensus_queue.cc:1035] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260808 20:14:48.706983 15674 raft_consensus.cc:3037] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:48.693291 15666 raft_consensus.cc:686] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:48.735240 12250 raft_consensus.cc:1261] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260808 20:14:48.732846 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 101341us
I20260808 20:14:48.700862 16061 raft_consensus.cc:3037] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:48.704523 16194 raft_consensus.cc:3032] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:48.706543 16014 raft_consensus.cc:3037] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:48.701107 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:48.711475 16090 raft_consensus.cc:729] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:48.705207 16201 raft_consensus.cc:2726] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:48.721251 15892 raft_consensus.cc:2781] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:48.640208 16049 consensus_queue.cc:237] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.777456 16090 consensus_queue.cc:260] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.778790 16090 raft_consensus.cc:3037] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:48.781198 16203 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:48.781733 16203 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:48.783805 16203 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:48.785755 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:14:48.786569 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:14:48.751694 16008 consensus_queue.cc:237] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:48.706334 13075 raft_consensus.cc:2454] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:48.794157 13079 raft_consensus.cc:1226] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:48.758544 12244 raft_consensus.cc:1226] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:48.733803 15142 consensus_queue.cc:1225] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer e5bd9a4616ce4acfa20dc7a002984a20 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260808 20:14:48.767900 15892 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:48.804757 15892 raft_consensus.cc:3037] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:48.747705 15835 raft_consensus.cc:2726] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 20:14:48.704041 16170 consensus_queue.cc:1046] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.111s
I20260808 20:14:48.748605 15762 consensus_queue.cc:1225] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer b9d94f35ea15421aa1d721201c782d8a log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260808 20:14:48.771543 16211 consensus_queue.cc:1035] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.021s
I20260808 20:14:48.773005 15951 consensus_queue.cc:1046] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260808 20:14:48.797431 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 101517us
I20260808 20:14:48.797643 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:48.803121 13074 raft_consensus.cc:1226] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 5. Current term is 6. Ops: [3.5-5.6]
I20260808 20:14:48.757697 16194 raft_consensus.cc:729] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:48.759588 15858 raft_consensus.cc:2726] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 20:14:48.813871 16152 log.cc:924] Time spent T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.600s user 0.002s sys 0.000s
I20260808 20:14:48.762403 16169 consensus_queue.cc:1035] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260808 20:14:48.818818 12001 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:48.719349 15776 consensus_queue.cc:237] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:48.825992 12244 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 7.
I20260808 20:14:48.759878 15602 consensus_queue.cc:1035] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.056s
I20260808 20:14:48.846320 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:48.830735 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 702507us
I20260808 20:14:48.847324 16064 raft_consensus.cc:3032] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:48.847680 16064 raft_consensus.cc:729] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:48.848667 11994 leader_election.cc:398] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Vote denied by peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) with higher term. Message: Invalid argument: T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5.
I20260808 20:14:48.848433 16064 consensus_queue.cc:260] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:48.849241 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:48.849608 16064 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:48.848986 11994 leader_election.cc:401] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260808 20:14:48.845700 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:48.789389 15666 consensus_queue.cc:237] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.846091 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:48.853525 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:48.854099 12141 raft_consensus.cc:3037] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:48.857236 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:48.858259 15973 raft_consensus.cc:3032] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:48.858637 15973 raft_consensus.cc:729] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:48.859395 15690 raft_consensus.cc:2781] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:48.859892 15690 raft_consensus.cc:686] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
W20260808 20:14:48.860608 16212 log.cc:924] Time spent T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20260808 20:14:48.861212 16093 raft_consensus.cc:3037] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:48.861714 11994 leader_election.cc:302] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:48.862399 16144 log.cc:924] Time spent T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.767s user 0.003s sys 0.000s
I20260808 20:14:48.863565 15858 raft_consensus.cc:482] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:48.863760 15997 raft_consensus.cc:2726] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:48.838856 15951 consensus_queue.cc:1225] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer e5bd9a4616ce4acfa20dc7a002984a20 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260808 20:14:48.863400 11994 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:48.866319 15997 raft_consensus.cc:2781] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 20:14:48.867461 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:48.865958 15949 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:48.876432 12137 raft_consensus.cc:2454] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 9.
I20260808 20:14:48.828212 16005 raft_consensus.cc:3032] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:48.812748 16176 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.883365 12101 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:48.892154 16196 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:48.892776 16196 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.897157 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:48.908766 12243 raft_consensus.cc:3032] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:48.902956 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:48.912048 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:48.909227 12243 raft_consensus.cc:729] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:48.918988 12147 raft_consensus.cc:1226] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:48.925349 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:48.881523 16098 raft_consensus.cc:482] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:48.932235 13016 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:48.932682 12102 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:48.918875 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:48.933542 13016 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:48.933576 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 179211us
I20260808 20:14:48.935729 12206 leader_election.cc:302] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:48.885665 16005 raft_consensus.cc:729] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:48.944702 13016 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:48.911809 15951 raft_consensus.cc:2781] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:14:48.952701 12206 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:48.955160 12000 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:48.864054 15858 raft_consensus.cc:504] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:48.911150 16196 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:48.924440 16200 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:48.961650 12141 raft_consensus.cc:2454] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 7.
I20260808 20:14:48.945355 15592 raft_consensus.cc:2726] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:48.962761 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:48.926656 13073 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 9 index: 11. (index mismatch)
I20260808 20:14:48.928468 12243 consensus_queue.cc:260] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.937570 15602 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:48.986168 12206 leader_election.cc:302] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:48.989300 12141 raft_consensus.cc:1261] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260808 20:14:48.949150 16163 raft_consensus.cc:2726] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 20:14:48.944571 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:48.860772 15690 consensus_queue.cc:237] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.952425 15951 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:48.953414 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:49.025764 15987 raft_consensus.cc:2726] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 20:14:49.025962 12138 raft_consensus.cc:2454] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:14:49.009795 15107 raft_consensus.cc:2781] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 20:14:49.011209 12152 raft_consensus.cc:1261] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260808 20:14:49.012778 15614 consensus_meta.cc:287] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.681s user 0.004s sys 0.005s
I20260808 20:14:48.968194 15835 raft_consensus.cc:2726] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 20:14:48.986589 15948 raft_consensus.cc:2726] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:49.014075 12206 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:48.999446 16203 raft_consensus.cc:482] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:48.998065 15142 consensus_queue.cc:1046] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.027s
I20260808 20:14:49.025070 15951 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:49.032606 16203 raft_consensus.cc:504] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:49.034718 16203 leader_election.cc:288] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:49.035363 15614 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:49.025449 16118 raft_consensus.cc:2726] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 20:14:49.036834 12047 raft_consensus.cc:2379] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 8: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:49.038874 15614 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:48.944231 16098 raft_consensus.cc:504] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:49.042783 16098 leader_election.cc:288] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:49.007417 16175 raft_consensus.cc:2726] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260808 20:14:49.047407 16143 log.cc:924] Time spent T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.879s user 0.002s sys 0.000s
I20260808 20:14:49.049834 15858 leader_election.cc:288] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:49.051535 16189 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:49.052062 16189 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:49.054371 16189 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:48.906816 15973 consensus_queue.cc:260] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:49.059934 15973 raft_consensus.cc:3037] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:49.064198 12243 raft_consensus.cc:3037] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:48.838269 16194 consensus_queue.cc:260] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:49.067793 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:49.068478 16176 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:49.073513 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 59525.8us
I20260808 20:14:48.990690 15892 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:49.077996 16014 raft_consensus.cc:2726] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 20:14:49.078161 12248 raft_consensus.cc:1261] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260808 20:14:49.079874 12144 raft_consensus.cc:2454] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
W20260808 20:14:49.008558 16152 log.cc:924] Time spent T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260808 20:14:49.080700 12045 raft_consensus.cc:2454] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 7.
W20260808 20:14:49.010718 16146 log.cc:924] Time spent T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260808 20:14:49.087071 12036 raft_consensus.cc:2454] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:14:49.095986 12206 leader_election.cc:302] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:49.112933 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 164047us
I20260808 20:14:48.970719 15886 consensus_queue.cc:1046] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.095s
I20260808 20:14:49.140870 12048 raft_consensus.cc:1226] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6. Ops: [3.3-5.4]
I20260808 20:14:49.141844 12205 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:49.162643 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:49.163297 12244 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 7
I20260808 20:14:49.167124 12205 leader_election.cc:302] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:49.182034 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:49.183036 13079 raft_consensus.cc:2454] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 7.
I20260808 20:14:49.184616 13080 raft_consensus.cc:1261] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260808 20:14:49.136786 15891 raft_consensus.cc:482] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:49.186962 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 104589us
I20260808 20:14:49.191236 12206 leader_election.cc:302] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:49.196446 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:49.197105 12036 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:49.006853 15602 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:49.182353 16225 log.cc:924] Time spent T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.245s user 0.002s sys 0.002s
I20260808 20:14:49.218544 12244 raft_consensus.cc:2454] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 7.
I20260808 20:14:49.202672 16202 raft_consensus.cc:482] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:49.165537 15142 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:49.182065 16118 raft_consensus.cc:2781] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:14:49.096868 16134 consensus_queue.cc:1046] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.109s
I20260808 20:14:49.185457 15614 raft_consensus.cc:3032] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Stepping down as leader of term 5
W20260808 20:14:49.232510 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 287865us
I20260808 20:14:49.279136 13016 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:49.065913 16005 consensus_queue.cc:260] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:48.993247 16200 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:49.240408 13014 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:49.237982 15949 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:49.209007 16175 raft_consensus.cc:2726] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:49.226863 15592 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:49.291405 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 51086.6us
I20260808 20:14:49.292623 12036 raft_consensus.cc:2454] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 7.
I20260808 20:14:49.232290 15142 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:49.232045 16202 raft_consensus.cc:504] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:49.303714 16207 raft_consensus.cc:2726] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 20:14:49.237776 16118 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:49.304399 16118 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:49.198233 15987 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.089s
W20260808 20:14:49.311088 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 358454us
W20260808 20:14:49.308732 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 344980us
I20260808 20:14:49.312345 12000 leader_election.cc:302] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:49.285182 15614 raft_consensus.cc:729] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:49.317206 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:49.320236 12040 raft_consensus.cc:2454] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:49.321139 16208 raft_consensus.cc:3032] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 7 LEADER]: Stepping down as leader of term 7
I20260808 20:14:49.223321 15835 consensus_queue.cc:1035] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.087s
I20260808 20:14:49.318506 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:49.321549 16208 raft_consensus.cc:729] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:49.322747 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:49.305183 15949 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:49.328420 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:49.304059 16202 leader_election.cc:288] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:49.303938 15142 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:49.201360 12206 leader_election.cc:302] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:49.336181 12241 raft_consensus.cc:2454] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:49.342463 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:49.310868 15972 raft_consensus.cc:2781] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 20:14:49.218317 16189 raft_consensus.cc:2781] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:49.375594 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:49.384374 12254 raft_consensus.cc:3037] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:49.314906 16196 raft_consensus.cc:2726] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:14:49.238903 15951 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:49.441917 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:49.336530 16008 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.014s sys 0.015s
I20260808 20:14:49.193495 16014 raft_consensus.cc:2726] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 20:14:49.388792 15826 raft_consensus.cc:482] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:49.430840 15972 raft_consensus.cc:686] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:49.375415 15949 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:49.394397 12101 leader_election.cc:302] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:49.396373 15858 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:49.397132 16205 consensus_queue.cc:1035] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.021s
I20260808 20:14:49.397431 15602 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:49.397763 16210 raft_consensus.cc:2726] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:49.398041 15886 raft_consensus.cc:482] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
W20260808 20:14:49.400318 16093 consensus_meta.cc:287] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.539s user 0.008s sys 0.000s
I20260808 20:14:49.496562 16242 raft_consensus.cc:2781] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:49.497153 16242 raft_consensus.cc:482] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:49.497823 16242 raft_consensus.cc:3037] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:49.497095 15886 raft_consensus.cc:504] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:49.073442 16049 consensus_queue.cc:1159] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b9d94f35ea15421aa1d721201c782d8a" responder_term: 5 status { last_received { term: 4 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 4. Current term is 5. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260808 20:14:49.406751 16234 log.cc:924] Time spent T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20260808 20:14:49.424058 16200 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:49.205171 15891 raft_consensus.cc:504] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:49.426836 15743 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.020s sys 0.033s
I20260808 20:14:49.309681 15592 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:49.477577 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 276286us
I20260808 20:14:49.424810 15614 consensus_queue.cc:260] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:49.392493 16203 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:49.392999 15892 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:49.551206 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:49.435222 15997 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.024s sys 0.019s
W20260808 20:14:49.486459 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 92125.8us
W20260808 20:14:49.487026 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 49339.8us
W20260808 20:14:49.480450 16008 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.687s user 0.014s sys 0.015s
I20260808 20:14:49.495391 16245 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:49.497386 16093 raft_consensus.cc:2726] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) with higher term. Message: Invalid argument: T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5.
I20260808 20:14:49.409600 12243 raft_consensus.cc:1261] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 20:14:49.488365 15142 raft_consensus.cc:2726] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260808 20:14:49.522755 15743 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.654s user 0.020s sys 0.033s
I20260808 20:14:49.514389 12254 raft_consensus.cc:2454] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:14:49.543606 12206 leader_election.cc:302] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:49.500955 16189 raft_consensus.cc:686] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:49.630816 16189 consensus_queue.cc:237] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:49.554518 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 274300us
I20260808 20:14:49.492290 15858 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:49.633693 16090 consensus_meta.cc:287] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.854s user 0.008s sys 0.004s
I20260808 20:14:49.634325 15886 leader_election.cc:288] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:49.635536 15858 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:49.488636 15826 raft_consensus.cc:504] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:49.638433 15826 leader_election.cc:288] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:49.594034 12046 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.775s user 0.004s sys 0.010s
W20260808 20:14:49.639098 12046 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.823s user 0.004s sys 0.010s
I20260808 20:14:49.640312 16188 raft_consensus.cc:482] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:49.641103 16188 raft_consensus.cc:504] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:49.644037 16188 leader_election.cc:288] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:49.605058 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
W20260808 20:14:49.556072 15997 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.634s user 0.024s sys 0.019s
W20260808 20:14:49.588485 15653 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.702s user 0.003s sys 0.011s
I20260808 20:14:49.487560 16208 consensus_queue.cc:260] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:49.645669 15997 raft_consensus.cc:686] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:49.646850 16208 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:49.647490 15997 consensus_queue.cc:237] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:49.649274 13078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 501) took 1110 ms. Trace:
W20260808 20:14:49.645772 15653 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.763s user 0.004s sys 0.012s
W20260808 20:14:49.650840 16235 log.cc:924] Time spent T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
I20260808 20:14:49.657204 16188 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:49.657833 16188 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:49.617455 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 73940us
I20260808 20:14:49.618515 15646 consensus_queue.cc:1035] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260808 20:14:49.610165 16121 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260808 20:14:49.661162 16242 raft_consensus.cc:504] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:49.620826 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:49.663429 12241 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 9.
I20260808 20:14:49.591074 16203 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:49.665380 16240 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:49.622054 15891 leader_election.cc:288] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:49.618350 15142 raft_consensus.cc:2726] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:49.623981 15951 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:49.624872 16228 log.cc:924] Time spent T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.150s user 0.001s sys 0.000s
I20260808 20:14:49.605937 15948 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:49.669144 15948 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:49.671885 15948 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:49.672966 15886 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:49.673799 15886 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:49.663424 15592 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:49.676606 15886 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:49.677910 12248 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 492) took 1044 ms. Trace:
I20260808 20:14:49.666180 16203 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:49.666260 16240 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:49.682482 13016 leader_election.cc:302] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:49.678848 16242 leader_election.cc:288] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:49.687343 15142 consensus_queue.cc:1035] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 20:14:49.688009 12206 leader_election.cc:302] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:49.687896 16102 consensus_queue.cc:1035] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
W20260808 20:14:49.689045 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 371748us
I20260808 20:14:49.686000 12248 rpcz_store.cc:276] 0808 20:14:48.633551 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:48.633672 (+ 121us) service_pool.cc:227] Handling call
0808 20:14:48.647749 (+ 14077us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:49.112857 (+465108us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:49.677663 (+564806us) spinlock_profiling.cc:243] Waited 430 ms on lock 0x7b5400142b54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:49.677845 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":989058944}
I20260808 20:14:49.692548 12000 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:49.649643 13078 rpcz_store.cc:276] 0808 20:14:48.539278 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:48.539397 (+ 119us) service_pool.cc:227] Handling call
0808 20:14:48.555134 (+ 15737us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:48.661863 (+106729us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:48.661894 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:49.194488 (+532594us) spinlock_profiling.cc:243] Waited 33.1 ms on lock 0x7b2c00091d88. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa966cbde11 00007fa966ccd442 00007fa968315c11 00007fa9682da58a 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772
0808 20:14:49.199163 (+ 4675us) log.cc:842] Serialized 26 byte log entry
0808 20:14:49.531988 (+332825us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:49.532022 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:49.532087 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:49.648960 (+116873us) raft_consensus.cc:1671] finished
0808 20:14:49.648978 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:49.649210 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32100,"raft.queue_time_us":44486,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":9100,"spinlock_wait_cycles":76051200,"thread_start_us":241418,"threads_started":2,"wal-append.queue_time_us":399814}
I20260808 20:14:49.696605 12000 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
W20260808 20:14:49.697546 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 76139us
I20260808 20:14:49.698220 12101 leader_election.cc:302] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:49.701856 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:49.702402 12036 raft_consensus.cc:1226] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 6. Current term is 7. Ops: []
I20260808 20:14:49.617103 15614 raft_consensus.cc:3037] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:49.669066 15949 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:49.706641 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 69518.1us
I20260808 20:14:49.710383 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:49.714174 12147 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.725s user 0.000s sys 0.026s
W20260808 20:14:49.714563 12147 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.747s user 0.000s sys 0.026s
I20260808 20:14:49.723206 12101 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:49.695194 16178 raft_consensus.cc:2781] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 20:14:49.725843 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:49.727577 12207 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:49.700989 16093 raft_consensus.cc:2726] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:49.731530 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:49.701277 16208 raft_consensus.cc:2726] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:49.730964 12150 raft_consensus.cc:1226] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7. Ops: []
I20260808 20:14:49.709038 16188 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:49.596463 15972 consensus_queue.cc:237] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:49.733381 12248 raft_consensus.cc:1226] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260808 20:14:49.733479 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:49.735070 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:49.705183 15690 raft_consensus.cc:2781] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:49.738523 12141 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 9 index: 11. (index mismatch)
I20260808 20:14:49.701208 15602 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:49.731727 12045 raft_consensus.cc:1261] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260808 20:14:49.766144 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:49.730006 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:49.755884 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:49.762224 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:49.763772 12241 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 9.
I20260808 20:14:49.819177 12150 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 8.
W20260808 20:14:49.820883 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 97741us
I20260808 20:14:49.810380 12047 raft_consensus.cc:2454] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:49.604072 16245 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:49.721312 15826 consensus_queue.cc:1035] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260808 20:14:49.721611 16090 consensus_queue.cc:1035] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260808 20:14:49.737762 15592 raft_consensus.cc:2726] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:49.817209 12244 raft_consensus.cc:1261] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260808 20:14:49.817580 12140 raft_consensus.cc:2396] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260808 20:14:49.732342 15858 raft_consensus.cc:2781] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:49.733237 16134 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:49.752511 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:49.813532 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:49.846884 12141 raft_consensus.cc:1226] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260808 20:14:49.849745 16178 raft_consensus.cc:686] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:49.764215 15690 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:14:49.809881 16228 log.cc:924] Time spent T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260808 20:14:49.860296 13072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53296 (request call id 735) took 1107 ms. Trace:
I20260808 20:14:49.813197 13075 raft_consensus.cc:1261] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 20:14:49.871704 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:49.810613 16273 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:49.814016 15602 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:50.084512 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 326564us
I20260808 20:14:50.099735 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:50.127250 13078 raft_consensus.cc:2379] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 5: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:49.948297 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1743) took 1056 ms. Trace:
I20260808 20:14:50.096385 12147 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1744) took 1199 ms. Trace:
W20260808 20:14:50.011245 16118 consensus_meta.cc:287] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.675s user 0.010s sys 0.000s
W20260808 20:14:50.099869 16250 log.cc:924] Time spent T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.470s user 0.000s sys 0.001s
I20260808 20:14:49.858011 15948 raft_consensus.cc:2781] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 20:14:50.128731 12147 rpcz_store.cc:276] 0808 20:14:48.897067 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:48.923269 (+ 26202us) service_pool.cc:227] Handling call
0808 20:14:48.923557 (+ 288us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:48.923737 (+ 180us) raft_consensus.cc:1520] Early marking committed up to index 4
0808 20:14:49.714921 (+791184us) log.cc:842] Serialized 14 byte log entry
0808 20:14:50.096034 (+381113us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:50.096090 (+ 56us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:50.096135 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:50.096203 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:50.096322 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":105389,"prepare.queue_time_us":631079,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":4344,"raft.queue_time_us":101162,"thread_start_us":1051836,"threads_started":2}
I20260808 20:14:50.133973 12254 raft_consensus.cc:1261] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260808 20:14:50.162880 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:49.847268 16188 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:49.957388 16266 consensus_queue.cc:1046] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.326s
I20260808 20:14:50.202009 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:50.257279 12137 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:14:50.128088 12146 rpcz_store.cc:276] 0808 20:14:48.891365 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:48.902624 (+ 11259us) service_pool.cc:227] Handling call
0808 20:14:48.927281 (+ 24657us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:48.927470 (+ 189us) raft_consensus.cc:1520] Early marking committed up to index 8
0808 20:14:49.329058 (+401588us) log.cc:842] Serialized 14 byte log entry
0808 20:14:49.692838 (+363780us) log.cc:842] Serialized 14 byte log entry
0808 20:14:49.722267 (+ 29429us) log.cc:842] Serialized 14 byte log entry
0808 20:14:49.947825 (+225558us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:49.947958 (+ 133us) raft_consensus.cc:1631] Marking committed up to 8
0808 20:14:49.948009 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:49.948084 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:49.948228 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":295436,"prepare.queue_time_us":686315,"raft.queue_time_us":267124,"raft.run_cpu_time_us":924,"raft.run_wall_time_us":44044,"spinlock_wait_cycles":114502016,"thread_start_us":820038,"threads_started":3,"wal-append.queue_time_us":485157}
I20260808 20:14:50.198303 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:50.259518 12140 raft_consensus.cc:2379] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:50.256353 12254 pending_rounds.cc:76] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a: Aborting all ops after (but not including) 2
I20260808 20:14:50.260245 12254 pending_rounds.cc:97] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260808 20:14:50.260586 12254 raft_consensus.cc:2866] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:50.197793 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:50.124454 16064 raft_consensus.cc:482] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:50.263780 12141 raft_consensus.cc:2379] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 8: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:49.969801 16259 raft_consensus.cc:482] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:50.019851 15835 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:50.127887 15690 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
W20260808 20:14:50.253875 16274 log.cc:924] Time spent T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.244s user 0.000s sys 0.000s
W20260808 20:14:50.203266 16277 log.cc:924] Time spent T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.141s user 0.000s sys 0.000s
I20260808 20:14:50.156177 16090 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:50.232352 16093 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:50.327095 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:50.167266 15826 raft_consensus.cc:482] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:50.130646 16203 raft_consensus.cc:482] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:50.296780 16271 raft_consensus.cc:482] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:50.198351 15873 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:50.203687 15891 raft_consensus.cc:482] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:50.170264 16061 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:50.172392 15997 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:50.476166 15873 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:50.498694 15997 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:50.167743 15142 raft_consensus.cc:482] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:50.472841 16240 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:50.501291 15142 raft_consensus.cc:504] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:50.502905 15873 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:50.503610 15873 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:50.258340 15948 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:50.250447 16245 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:50.296037 15886 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:50.507020 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:50.504487 15948 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Advancing to term 10
I20260808 20:14:50.335929 15107 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:50.514580 15107 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 FOLLOWER]: Advancing to term 10
I20260808 20:14:50.308152 16064 raft_consensus.cc:504] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:50.247208 16178 consensus_queue.cc:237] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:50.524116 15891 raft_consensus.cc:504] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:50.527304 16284 log.cc:924] Time spent T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.262s user 0.000s sys 0.000s
I20260808 20:14:50.305320 16188 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:50.534718 15602 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:50.205606 15743 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:50.331271 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:50.535295 15602 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:50.535691 12254 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8.
I20260808 20:14:50.535737 15602 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:50.536957 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:50.471061 11897 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44508 (request call id 193) took 22863 ms (client timeout 15000 ms). Trace:
I20260808 20:14:50.537112 16061 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:50.469477 15934 raft_consensus.cc:3032] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 20:14:50.538406 15934 raft_consensus.cc:729] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
W20260808 20:14:50.498144 16274 log.cc:924] Time spent T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.148s user 0.000s sys 0.001s
I20260808 20:14:50.539594 15934 consensus_queue.cc:260] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:50.190114 16118 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:50.153949 16134 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:50.504431 15873 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:50.595467 12150 raft_consensus.cc:1261] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 20:14:50.153764 16273 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:50.646788 12144 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:50.335721 15835 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:50.504127 15142 leader_election.cc:288] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:50.542196 15743 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:50.537671 11897 rpcz_store.cc:269] 0808 20:14:27.607501 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:27.607718 (+ 217us) service_pool.cc:227] Handling call
0808 20:14:27.761743 (+154025us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b5400002dd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825faf0 00007fa9682e2f2a 00007fa96e234d6c 00007fa96e32ec3e 00007fa96c640fb4 00007fa96c640f42 00007fa96c640e71 00007fa96c640dec 00007fa96c6400a2 00007fa96670494c 00007fa966703d86
0808 20:14:50.470963 (+22709220us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 20:14:47.755145 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 20:14:47.814727 (+ 59582us) write_op.cc:432] Acquiring schema lock in shared mode
0808 20:14:47.814781 (+ 54us) write_op.cc:435] Acquired schema lock
0808 20:14:47.814803 (+ 22us) tablet.cc:660] Decoding operations
0808 20:14:47.815221 (+ 418us) write_op.cc:620] Acquiring the partition lock for write op
0808 20:14:47.815308 (+ 87us) write_op.cc:641] Partition lock acquired for write op
0808 20:14:47.815333 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0808 20:14:47.815568 (+ 235us) tablet.cc:699] Row locks acquired
0808 20:14:47.815608 (+ 40us) write_op.cc:260] PREPARE: finished
0808 20:14:47.815790 (+ 182us) write_op.cc:270] Start()
0808 20:14:47.815924 (+ 134us) write_op.cc:276] Timestamp: P: 1786220087815722 usec, L: 0
0808 20:14:47.815955 (+ 31us) op_driver.cc:331] REPLICATION: starting
0808 20:14:47.816608 (+ 653us) log.cc:842] Serialized 530 byte log entry
0808 20:14:48.272032 (+455424us) spinlock_profiling.cc:243] Waited 94.6 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:48.532113 (+260081us) op_driver.cc:444] REPLICATION: finished
0808 20:14:48.828494 (+296381us) write_op.cc:301] APPLY: starting
0808 20:14:48.842161 (+ 13667us) tablet.cc:1365] starting BulkCheckPresence
0808 20:14:48.907110 (+ 64949us) tablet.cc:1368] finished BulkCheckPresence
0808 20:14:48.913824 (+ 6714us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 20:14:49.036949 (+123125us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 20:14:49.037193 (+ 244us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0808 20:14:49.037238 (+ 45us) write_op.cc:312] APPLY: finished
0808 20:14:49.037501 (+ 263us) log.cc:842] Serialized 23 byte log entry
0808 20:14:49.037677 (+ 176us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:14:49.037757 (+ 80us) write_op.cc:454] Released schema lock
0808 20:14:49.037810 (+ 53us) write_op.cc:341] FINISH: Updating metrics
0808 20:14:49.038090 (+ 280us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:14:49.038115 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":997999,"reported_tablets":20,"spinlock_wait_cycles":327576320,"child_traces":[["op",{"apply.queue_time_us":211361,"apply.run_cpu_time_us":2441,"apply.run_wall_time_us":221940,"num_ops":1,"prepare.queue_time_us":525487,"prepare.run_cpu_time_us":22310,"prepare.run_wall_time_us":260116,"raft.queue_time_us":107779,"raft.run_cpu_time_us":17258,"raft.run_wall_time_us":286258,"replication_time_us":705233,"spinlock_wait_cycles":217574656,"thread_start_us":670813,"threads_started":4,"wal-append.queue_time_us":233171,"wal-append.run_cpu_time_us":25379,"wal-append.run_wall_time_us":1989418}]]}
I20260808 20:14:50.465230 16093 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:50.470299 15826 raft_consensus.cc:504] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:50.473331 16203 raft_consensus.cc:504] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:50.717525 12207 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.554s user 0.010s sys 0.042s
W20260808 20:14:50.719317 12207 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.556s user 0.010s sys 0.042s
W20260808 20:14:50.717970 12000 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.011s sys 0.040s
W20260808 20:14:50.721271 12000 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.540s user 0.012s sys 0.040s
W20260808 20:14:50.721642 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 540678us
I20260808 20:14:50.466622 16090 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:50.719957 12150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1465) took 1007 ms. Trace:
I20260808 20:14:50.669281 16273 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:50.745599 12150 rpcz_store.cc:276] 0808 20:14:49.712660 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:49.820765 (+108105us) service_pool.cc:227] Handling call
0808 20:14:50.556006 (+735241us) spinlock_profiling.cc:243] Waited 26.2 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:50.556232 (+ 226us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:50.644760 (+ 88528us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:50.719888 (+ 75128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":60285696}
I20260808 20:14:50.558331 15997 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:50.307646 16259 raft_consensus.cc:504] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:50.688760 16188 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:50.764339 12147 raft_consensus.cc:2454] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:50.585290 16064 leader_election.cc:288] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:50.757508 12102 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.027s sys 0.027s
W20260808 20:14:50.772678 12102 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.647s user 0.027s sys 0.028s
W20260808 20:14:50.772989 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 647658us
W20260808 20:14:50.766206 12243 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.000s sys 0.041s
W20260808 20:14:50.769965 12101 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.647s user 0.026s sys 0.036s
W20260808 20:14:50.774394 12101 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.651s user 0.026s sys 0.036s
W20260808 20:14:50.774724 12243 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.571s user 0.000s sys 0.042s
I20260808 20:14:50.475649 16271 raft_consensus.cc:504] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:50.741925 15934 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Advancing to term 7
W20260808 20:14:50.747079 15858 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.001s sys 0.018s
I20260808 20:14:50.553465 15886 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:50.752743 15891 leader_election.cc:288] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:50.826224 12047 raft_consensus.cc:2396] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 6 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260808 20:14:50.766479 16061 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:50.781136 16273 raft_consensus.cc:482] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:50.805665 15858 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.638s user 0.001s sys 0.018s
I20260808 20:14:50.783478 15997 raft_consensus.cc:482] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:50.851313 13015 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.726s user 0.026s sys 0.024s
W20260808 20:14:50.854813 13015 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.729s user 0.027s sys 0.024s
W20260808 20:14:50.855120 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 729527us
I20260808 20:14:50.856067 13015 leader_election.cc:302] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:50.787040 16245 raft_consensus.cc:482] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:50.861174 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:50.862434 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:50.863078 12146 raft_consensus.cc:2454] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:50.863826 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:50.864403 12146 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 7.
I20260808 20:14:50.873350 13016 leader_election.cc:302] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:50.854425 15858 raft_consensus.cc:686] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:50.880005 15858 consensus_queue.cc:237] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:50.885241 13079 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.986s user 0.003s sys 0.025s
W20260808 20:14:50.886004 13079 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.986s user 0.004s sys 0.025s
W20260808 20:14:50.885632 12105 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.132s user 0.010s sys 0.019s
W20260808 20:14:50.886740 12105 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.133s user 0.010s sys 0.019s
I20260808 20:14:50.831915 15891 raft_consensus.cc:482] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:50.904455 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
W20260808 20:14:50.862247 15614 consensus_meta.cc:287] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 1.153s user 0.005s sys 0.008s
I20260808 20:14:50.887209 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:50.906221 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:50.907948 12036 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 7.
W20260808 20:14:50.911659 12247 consensus_meta.cc:287] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.784s user 0.010s sys 0.000s
I20260808 20:14:50.919490 12247 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 9.
I20260808 20:14:50.920140 12247 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1840) took 1166 ms. Trace:
I20260808 20:14:50.920501 12247 rpcz_store.cc:276] 0808 20:14:49.753303 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:49.753438 (+ 135us) service_pool.cc:227] Handling call
0808 20:14:50.920052 (+1166614us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:50.921756 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:50.922580 12247 raft_consensus.cc:2379] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:50.923537 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:50.925573 12203 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.721s user 0.028s sys 0.036s
W20260808 20:14:50.926004 12203 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.763s user 0.028s sys 0.036s
I20260808 20:14:50.858177 16061 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:50.859891 16064 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:50.865067 15602 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:50.934275 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:50.936443 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:50.937175 12140 raft_consensus.cc:2454] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:14:50.937414 13074 raft_consensus.cc:1226] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260808 20:14:50.918853 15690 consensus_meta.cc:287] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.584s user 0.008s sys 0.000s
I20260808 20:14:50.930799 13077 raft_consensus.cc:2379] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 6: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:50.906209 12150 raft_consensus.cc:1261] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260808 20:14:50.931142 13014 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.825s user 0.017s sys 0.042s
W20260808 20:14:50.918893 11994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.008s sys 0.016s
W20260808 20:14:50.946965 13014 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.841s user 0.017s sys 0.042s
W20260808 20:14:50.947260 11994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.592s user 0.008s sys 0.017s
W20260808 20:14:50.947655 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 592340us
I20260808 20:14:50.940237 16320 consensus_queue.cc:1035] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:50.933451 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:50.877053 16245 raft_consensus.cc:504] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:50.939252 16298 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:50.939855 13018 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.823s user 0.000s sys 0.028s
W20260808 20:14:50.957650 13018 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.901s user 0.000s sys 0.029s
I20260808 20:14:50.864614 15997 raft_consensus.cc:504] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:50.941435 16306 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:50.941675 16090 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:50.939682 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:50.943931 13073 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 506) took 1231 ms. Trace:
I20260808 20:14:50.866770 16273 raft_consensus.cc:504] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:50.935540 15614 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:50.939154 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:50.927919 12206 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.008s sys 0.035s
I20260808 20:14:50.947402 16064 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:50.929797 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:50.987151 12038 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Advancing to term 8
W20260808 20:14:50.987341 12206 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.649s user 0.008s sys 0.035s
I20260808 20:14:50.953838 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:50.927752 16291 raft_consensus.cc:482] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:50.940749 16297 raft_consensus.cc:482] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:50.962540 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:50.940100 16265 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:50.939728 16311 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:50.953699 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:50.945025 16308 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:50.952524 15602 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:50.989835 12247 raft_consensus.cc:2454] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 8.
I20260808 20:14:50.987664 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:50.927424 15891 raft_consensus.cc:504] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:50.999789 12036 raft_consensus.cc:2379] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 4: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
W20260808 20:14:50.963193 16189 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.707s user 0.009s sys 0.005s
I20260808 20:14:50.937702 16061 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:50.966812 13073 rpcz_store.cc:276] 0808 20:14:49.712685 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:49.723473 (+ 10788us) service_pool.cc:227] Handling call
0808 20:14:49.822420 (+ 98947us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:50.049267 (+226847us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:50.049337 (+ 70us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:50.049993 (+ 656us) log.cc:842] Serialized 26 byte log entry
0808 20:14:50.331355 (+281362us) raft_consensus.cc:1631] Marking committed up to 9
0808 20:14:50.331395 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:50.331463 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:50.943727 (+612264us) raft_consensus.cc:1671] finished
0808 20:14:50.943748 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:50.943874 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54590,"raft.queue_time_us":170529,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":32181,"thread_start_us":276555,"threads_started":1,"wal-append.queue_time_us":772867}
I20260808 20:14:50.997376 16090 consensus_queue.cc:1035] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260808 20:14:50.986128 15873 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:50.990185 15835 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:50.965812 16317 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:50.998474 15743 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:50.949198 15690 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:51.012598 16189 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.757s user 0.009s sys 0.005s
I20260808 20:14:51.003520 16093 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:50.999502 16118 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:51.045342 16303 log.cc:924] Time spent T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.112s user 0.001s sys 0.000s
I20260808 20:14:51.045653 16118 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:50.125504 13072 rpcz_store.cc:276] 0808 20:14:48.753084 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:48.759736 (+ 6652us) service_pool.cc:227] Handling call
0808 20:14:48.813185 (+ 53449us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 20:14:48.813659 (+ 474us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:48.813697 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 20:14:48.814307 (+ 610us) log.cc:842] Serialized 52 byte log entry
0808 20:14:49.067156 (+252849us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:49.715504 (+648348us) log.cc:842] Serialized 14 byte log entry
0808 20:14:49.758894 (+ 43390us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:49.815066 (+ 56172us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:49.823044 (+ 7978us) raft_consensus.cc:1671] finished
0808 20:14:49.823061 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:49.860217 (+ 37156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35896,"prepare.queue_time_us":1002806,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":80654,"raft.run_cpu_time_us":323,"raft.run_wall_time_us":7739,"thread_start_us":705106,"threads_started":2,"wal-append.queue_time_us":374670}
I20260808 20:14:51.016530 16201 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:51.025300 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:51.050015 12247 raft_consensus.cc:3037] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:51.043318 16064 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:51.007535 16308 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:50.987344 16298 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:50.997865 16321 raft_consensus.cc:2781] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:51.009021 15826 leader_election.cc:288] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:50.998229 15614 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:51.056196 12038 raft_consensus.cc:2454] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 8.
I20260808 20:14:51.013803 16203 leader_election.cc:288] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:51.141494 12045 raft_consensus.cc:1226] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7. Ops: []
I20260808 20:14:51.023307 15873 raft_consensus.cc:482] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
W20260808 20:14:51.021659 16208 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.668s user 0.025s sys 0.037s
W20260808 20:14:51.071465 16178 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.006s sys 0.018s
I20260808 20:14:51.008762 16187 consensus_queue.cc:1035] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.283s
I20260808 20:14:51.187012 12205 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:51.025837 16220 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:51.012918 15602 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:51.050104 16134 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:51.198433 12150 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
W20260808 20:14:51.123234 12000 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.796s (SENT) [suppressed 31 similar messages]
I20260808 20:14:51.052989 16259 leader_election.cc:288] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:51.137256 16321 raft_consensus.cc:482] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:14:51.213179 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 89981.7us
I20260808 20:14:50.997468 16291 raft_consensus.cc:504] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:51.190640 16208 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.847s user 0.028s sys 0.040s
I20260808 20:14:51.049324 16118 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:51.139465 15835 raft_consensus.cc:482] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
W20260808 20:14:51.201340 16178 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.671s user 0.007s sys 0.018s
I20260808 20:14:50.989356 16306 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:51.281132 12140 raft_consensus.cc:2454] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:14:51.191006 16304 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
W20260808 20:14:51.236749 11995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.009s sys 0.017s
I20260808 20:14:51.185123 16327 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:51.202390 16064 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:51.231228 15972 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.008s sys 0.013s
I20260808 20:14:51.005128 16265 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:51.208562 15826 raft_consensus.cc:482] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:51.142199 16189 consensus_queue.cc:1046] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.096s
I20260808 20:14:51.201527 16271 leader_election.cc:288] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:50.999341 16297 raft_consensus.cc:504] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:51.005232 16311 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:51.023536 16317 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:51.195492 15873 raft_consensus.cc:504] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:51.064779 16201 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:51.752561 11995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.417s user 0.016s sys 0.030s
I20260808 20:14:51.356993 12247 raft_consensus.cc:2454] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
W20260808 20:14:51.536065 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 320852us
I20260808 20:14:51.330828 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:14:51.216396 16134 raft_consensus.cc:482] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:51.287431 13014 leader_election.cc:302] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:51.757002 12000 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:51.290360 15835 raft_consensus.cc:504] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:51.756987 16178 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:51.753985 12247 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2005) took 1007 ms. Trace:
I20260808 20:14:51.758276 16178 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:51.758786 12247 rpcz_store.cc:276] 0808 20:14:50.746952 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:51.011745 (+264793us) service_pool.cc:227] Handling call
0808 20:14:51.753922 (+742177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:51.759788 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:51.760519 16178 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:51.765219 12105 leader_election.cc:302] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:51.769309 12000 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
W20260808 20:14:51.773314 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 485943us
W20260808 20:14:51.752516 15972 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.103s user 0.031s sys 0.093s
I20260808 20:14:51.216998 16326 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:51.480537 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 489875us
W20260808 20:14:51.796036 16328 log.cc:924] Time spent T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.603s user 0.000s sys 0.001s
W20260808 20:14:51.796692 15948 consensus_meta.cc:287] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 1.288s user 0.009s sys 0.006s
I20260808 20:14:51.797917 15873 leader_election.cc:288] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:51.503883 16259 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:51.205781 16220 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:51.809866 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:52.180817 12206 leader_election.cc:302] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:52.047956 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:52.142263 12040 consensus_meta.cc:287] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 1.214s user 0.011s sys 0.000s
I20260808 20:14:51.813220 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:51.499082 16187 raft_consensus.cc:482] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:52.184923 12040 raft_consensus.cc:2454] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:14:51.716531 16182 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:51.747484 16312 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:51.756174 16271 raft_consensus.cc:482] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:52.190848 15948 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:51.756322 15826 raft_consensus.cc:504] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:51.778676 16194 raft_consensus.cc:3037] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:52.194288 16369 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:52.195600 16375 raft_consensus.cc:482] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
W20260808 20:14:52.196081 12152 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1458) took 2486 ms (client timeout 3018 ms). Trace:
I20260808 20:14:52.196908 16371 raft_consensus.cc:2781] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader election won for term 8
I20260808 20:14:51.584497 15886 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:51.807744 16360 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:51.953219 15690 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:52.199656 12141 raft_consensus.cc:1226] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:51.616771 16273 leader_election.cc:288] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:51.584116 15891 leader_election.cc:288] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:52.219570 12140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1762) took 1346 ms. Trace:
I20260808 20:14:52.241585 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 397) took 3370 ms. Trace:
I20260808 20:14:51.217391 16321 raft_consensus.cc:3037] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
W20260808 20:14:52.022737 16330 log.cc:924] Time spent T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.499s user 0.001s sys 0.001s
W20260808 20:14:52.059554 15107 consensus_meta.cc:287] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 1.545s user 0.013s sys 0.000s
I20260808 20:14:51.499423 16245 leader_election.cc:288] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:51.359110 16322 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
W20260808 20:14:51.801034 16315 log.cc:924] Time spent T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.388s user 0.000s sys 0.001s
W20260808 20:14:51.807284 16319 log.cc:924] Time spent T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.158s user 0.000s sys 0.000s
I20260808 20:14:51.942147 16064 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:51.482911 15997 leader_election.cc:288] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:52.257114 12152 rpcz_store.cc:269] 0808 20:14:49.709753 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:49.711179 (+ 1426us) service_pool.cc:227] Handling call
0808 20:14:52.195820 (+2484641us) spinlock_profiling.cc:243] Waited 2.01 s on lock 0x7b5400017054. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:52.195978 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4626577664}
I20260808 20:14:51.810163 15835 leader_election.cc:288] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:51.988548 15602 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:51.803743 16352 log.cc:924] Time spent T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20260808 20:14:52.198055 16343 raft_consensus.cc:482] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:52.199316 16201 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:51.788128 16061 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:51.803611 16332 raft_consensus.cc:482] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:52.259398 13079 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.615s user 0.000s sys 0.012s
I20260808 20:14:52.173897 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:52.269892 12207 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.061s user 0.002s sys 0.015s
W20260808 20:14:52.360738 12207 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.152s user 0.002s sys 0.015s
I20260808 20:14:51.798653 16297 leader_election.cc:288] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:51.805800 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:52.010667 16265 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:52.362928 13072 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
I20260808 20:14:51.811635 16306 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:51.808497 16298 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:14:52.250586 13075 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.770s user 0.005s sys 0.011s
I20260808 20:14:52.184939 12206 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
W20260808 20:14:52.371141 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 186285us
I20260808 20:14:52.363147 16265 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 20:14:52.207322 15873 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:52.273128 16371 raft_consensus.cc:686] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 8 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:52.266958 16271 raft_consensus.cc:504] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:52.279363 15948 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 10 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:52.026147 16311 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:51.808844 16308 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:14:52.370606 13075 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.890s user 0.005s sys 0.011s
I20260808 20:14:52.281638 12140 rpcz_store.cc:276] 0808 20:14:50.872856 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:50.938429 (+ 65573us) service_pool.cc:227] Handling call
0808 20:14:52.219347 (+1280918us) spinlock_profiling.cc:243] Waited 290 ms on lock 0x7b5400060754. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:52.219462 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":668145920}
I20260808 20:14:52.221560 15614 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:52.015908 16357 raft_consensus.cc:2781] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:51.755311 16304 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:52.294659 15826 leader_election.cc:288] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:52.022229 15699 consensus_queue.cc:1046] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.249s
I20260808 20:14:51.809643 16118 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:52.280962 12142 rpcz_store.cc:276] 0808 20:14:48.870984 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:48.871401 (+ 417us) service_pool.cc:227] Handling call
0808 20:14:48.904705 (+ 33304us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:48.905096 (+ 391us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:48.905127 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:50.717585 (+1812458us) spinlock_profiling.cc:243] Waited 2.61 ms on lock 0x7b2c00000238. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa966cbde11 00007fa966ccd442 00007fa968315c11 00007fa9682da58a 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772
0808 20:14:50.745210 (+ 27625us) log.cc:842] Serialized 26 byte log entry
0808 20:14:51.206112 (+460902us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:51.206147 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:51.206218 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:52.222071 (+1015853us) raft_consensus.cc:1671] finished
0808 20:14:52.222093 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:52.241493 (+ 19400us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":111,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":12237,"spinlock_wait_cycles":6004096,"thread_start_us":416797,"threads_started":1,"wal-append.queue_time_us":1445231}
I20260808 20:14:52.258553 16369 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:51.780285 16134 raft_consensus.cc:504] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:51.981457 16203 consensus_queue.cc:1035] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.290s
I20260808 20:14:52.192193 16372 raft_consensus.cc:2781] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:52.207816 16259 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:52.273581 16375 raft_consensus.cc:504] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:52.334371 16270 log.cc:924] Time spent T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 2.504s user 0.006s sys 0.009s
I20260808 20:14:52.335846 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:52.290091 15891 raft_consensus.cc:2726] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:14:52.270169 16187 raft_consensus.cc:504] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:52.296423 15107 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:52.301120 16207 consensus_queue.cc:1035] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260808 20:14:52.573454 12249 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 500) took 3476 ms. Trace:
I20260808 20:14:52.342656 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:52.296021 16220 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:52.343741 12150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1760) took 1471 ms. Trace:
I20260808 20:14:52.246448 16182 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:52.346376 16121 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:52.347249 12147 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1753) took 2522 ms (client timeout 2152 ms). Trace:
W20260808 20:14:52.348177 12047 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1517) took 1841 ms (client timeout 1544 ms). Trace:
I20260808 20:14:52.351238 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 401) took 1489 ms. Trace:
I20260808 20:14:51.778405 16327 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:51.800622 16317 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:52.363365 16355 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
W20260808 20:14:52.365517 15934 consensus_meta.cc:287] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 1.557s user 0.005s sys 0.010s
I20260808 20:14:52.281092 16360 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:52.214821 16326 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:52.439963 13079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53290 (request call id 630) took 2958 ms. Trace:
I20260808 20:14:52.361061 16201 raft_consensus.cc:2726] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:52.524685 16359 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
W20260808 20:14:52.528573 12102 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.129s user 0.024s sys 0.016s
W20260808 20:14:52.532636 12205 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.322s user 0.002s sys 0.020s
I20260808 20:14:52.574390 12244 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Advancing to term 8
W20260808 20:14:52.624791 12205 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.416s user 0.002s sys 0.020s
W20260808 20:14:52.625207 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 1.4391e+06us
I20260808 20:14:52.629261 16326 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:52.558148 16372 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:52.550374 13075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53296 (request call id 742) took 1645 ms. Trace:
I20260808 20:14:52.478992 15873 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:52.395846 16336 raft_consensus.cc:2726] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:52.533506 16357 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:52.246694 16312 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:52.350289 16343 raft_consensus.cc:504] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:52.546093 16306 raft_consensus.cc:2781] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 20:14:52.564376 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:52.566340 13074 raft_consensus.cc:2454] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:52.563788 16118 raft_consensus.cc:482] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:52.302492 16322 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:52.630616 12102 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.231s user 0.024s sys 0.017s
I20260808 20:14:52.576457 16265 raft_consensus.cc:482] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:52.690725 12152 raft_consensus.cc:2362] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5. Re-sending same reply.
I20260808 20:14:52.632282 12150 rpcz_store.cc:276] 0808 20:14:50.872366 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:50.947309 (+ 74943us) service_pool.cc:227] Handling call
0808 20:14:52.343552 (+1396243us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b540017f054. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:52.343677 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":467468032}
I20260808 20:14:52.556726 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:52.575131 16134 leader_election.cc:288] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:52.573827 12249 rpcz_store.cc:276] 0808 20:14:49.097012 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:49.097171 (+ 159us) service_pool.cc:227] Handling call
0808 20:14:49.162030 (+ 64859us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:49.187712 (+ 25682us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:49.187742 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:49.266546 (+ 78804us) log.cc:842] Serialized 26 byte log entry
0808 20:14:49.538674 (+272128us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:49.538712 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:49.538777 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:52.317812 (+2779035us) spinlock_profiling.cc:243] Waited 3.31 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:52.573065 (+255253us) raft_consensus.cc:1671] finished
0808 20:14:52.573087 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:52.573340 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55766,"raft.queue_time_us":38007,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":18500,"spinlock_wait_cycles":7602432,"thread_start_us":223652,"threads_started":1,"wal-append.queue_time_us":408385}
W20260808 20:14:52.632567 12047 rpcz_store.cc:269] 0808 20:14:50.506574 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:50.506737 (+ 163us) service_pool.cc:227] Handling call
0808 20:14:52.347890 (+1841153us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x7b5400170554. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:52.348049 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":732007168}
W20260808 20:14:52.632663 12147 rpcz_store.cc:269] 0808 20:14:49.825201 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:50.265177 (+439976us) service_pool.cc:227] Handling call
0808 20:14:52.347072 (+2081895us) spinlock_profiling.cc:243] Waited 412 ms on lock 0x7b54003ec954. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:52.347180 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":947758464}
I20260808 20:14:52.629431 16121 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:52.637048 12146 rpcz_store.cc:276] 0808 20:14:50.861457 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:50.865133 (+ 3676us) service_pool.cc:227] Handling call
0808 20:14:50.865398 (+ 265us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:51.005533 (+140135us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:51.005564 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:51.005605 (+ 41us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:14:51.005634 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:52.351006 (+1345372us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b5400106654. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:52.351051 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:52.351176 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":318769920}
I20260808 20:14:52.368039 16332 raft_consensus.cc:504] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:52.634810 13075 rpcz_store.cc:276] 0808 20:14:50.905269 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:50.907682 (+ 2413us) service_pool.cc:227] Handling call
0808 20:14:50.956886 (+ 49204us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:50.994802 (+ 37916us) raft_consensus.cc:1520] Early marking committed up to index 8
0808 20:14:51.335475 (+340673us) log.cc:842] Serialized 14 byte log entry
0808 20:14:52.550032 (+1214557us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:52.550091 (+ 59us) raft_consensus.cc:1631] Marking committed up to 8
0808 20:14:52.550138 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:52.550201 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:52.550318 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7649,"prepare.queue_time_us":1062165,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":14941,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":2157,"thread_start_us":1432997,"threads_started":2}
I20260808 20:14:52.629907 13079 rpcz_store.cc:276] 0808 20:14:49.481917 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:49.482082 (+ 165us) service_pool.cc:227] Handling call
0808 20:14:49.483644 (+ 1562us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:49.503472 (+ 19828us) raft_consensus.cc:1520] Early marking committed up to index 8
0808 20:14:49.811121 (+307649us) log.cc:842] Serialized 14 byte log entry
0808 20:14:51.498010 (+1686889us) log.cc:842] Serialized 14 byte log entry
0808 20:14:52.350903 (+852893us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:52.364596 (+ 13693us) log.cc:842] Serialized 26 byte log entry
0808 20:14:52.390630 (+ 26034us) raft_consensus.cc:1631] Marking committed up to 8
0808 20:14:52.390667 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:52.390737 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:52.439731 (+ 48994us) raft_consensus.cc:1671] finished
0808 20:14:52.439752 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:52.439879 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10425,"prepare.queue_time_us":1465296,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":67,"raft.queue_time_us":940906,"raft.run_cpu_time_us":574,"raft.run_wall_time_us":44445,"thread_start_us":2565571,"threads_started":5,"wal-append.queue_time_us":1298786}
W20260808 20:14:52.565029 16361 log.cc:924] Time spent T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.236s user 0.001s sys 0.000s
I20260808 20:14:52.659613 16360 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:52.551198 16308 consensus_queue.cc:1035] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.085s
I20260808 20:14:52.702527 16322 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:52.703217 16322 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:52.572702 16375 leader_election.cc:288] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:52.661087 16359 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:52.671499 16187 leader_election.cc:288] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:52.675640 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:52.718617 12245 raft_consensus.cc:2379] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:52.676698 12045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1518) took 2163 ms. Trace:
I20260808 20:14:52.718120 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:52.719349 12249 raft_consensus.cc:2454] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:14:52.677398 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
W20260808 20:14:52.719909 12249 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2003) took 1980 ms (client timeout 2053 ms). Trace:
I20260808 20:14:52.683322 15107 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 10 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:52.634599 16201 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:52.639958 16372 raft_consensus.cc:3037] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
W20260808 20:14:52.691835 12142 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1763) took 1568 ms (client timeout 1888 ms). Trace:
W20260808 20:14:52.747603 12245 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1845) took 2729 ms (client timeout 1920 ms). Trace:
W20260808 20:14:52.691903 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 1.2929e+06us
I20260808 20:14:52.703111 16343 leader_election.cc:288] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:52.633724 16355 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:52.554563 15826 raft_consensus.cc:3032] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 LEADER]: Stepping down as leader of term 4
W20260808 20:14:52.694335 12140 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1768) took 1551 ms (client timeout 1814 ms). Trace:
I20260808 20:14:52.700634 16064 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:52.753055 12140 rpcz_store.cc:269] 0808 20:14:51.142729 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:52.543134 (+1400405us) service_pool.cc:227] Handling call
0808 20:14:52.694275 (+151141us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:52.532788 12138 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1459) took 2822 ms (client timeout 2509 ms). Trace:
I20260808 20:14:52.698329 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:52.663651 16357 raft_consensus.cc:3037] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:52.693356 16118 raft_consensus.cc:504] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:52.702258 16332 leader_election.cc:288] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:52.685696 16306 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:52.650936 16369 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:14:52.705854 13015 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.492s user 0.020s sys 0.041s
I20260808 20:14:52.662388 16371 consensus_queue.cc:237] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:52.765132 13015 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.564s user 0.021s sys 0.041s
W20260808 20:14:52.720258 12249 rpcz_store.cc:269] 0808 20:14:50.739052 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:52.699025 (+1959973us) service_pool.cc:227] Handling call
0808 20:14:52.719858 (+ 20833us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:52.720711 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:52.698671 16265 raft_consensus.cc:504] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:52.722769 11994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.822s user 0.003s sys 0.015s
I20260808 20:14:52.719095 12045 rpcz_store.cc:276] 0808 20:14:50.513283 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:50.513437 (+ 154us) service_pool.cc:227] Handling call
0808 20:14:50.513903 (+ 466us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:51.143368 (+629465us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:52.676416 (+1533048us) spinlock_profiling.cc:243] Waited 627 ms on lock 0x7b540038f4d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:52.676612 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1442922368}
W20260808 20:14:52.770818 11994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.870s user 0.003s sys 0.015s
I20260808 20:14:52.747275 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:52.747869 12142 rpcz_store.cc:269] 0808 20:14:51.123839 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:52.548934 (+1425095us) service_pool.cc:227] Handling call
0808 20:14:52.691775 (+142841us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:52.747993 12245 rpcz_store.cc:269] 0808 20:14:49.993573 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:49.993841 (+ 268us) service_pool.cc:227] Handling call
0808 20:14:52.675453 (+2681612us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b58001a1398. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:52.722889 (+ 47436us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23257344}
I20260808 20:14:52.700668 16134 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
W20260808 20:14:52.751828 12241 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2000) took 2925 ms (client timeout 2137 ms). Trace:
I20260808 20:14:52.695410 12244 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8.
I20260808 20:14:52.703938 16322 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:52.757762 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:52.759294 12146 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
I20260808 20:14:52.766261 12147 raft_consensus.cc:2454] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:14:52.721529 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:52.764487 16259 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:52.529603 12105 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.763s user 0.015s sys 0.020s
I20260808 20:14:52.774371 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:52.910204 12105 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.144s user 0.015s sys 0.020s
I20260808 20:14:52.764382 16306 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:14:52.757253 15826 raft_consensus.cc:729] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
W20260808 20:14:52.899677 16331 log.cc:924] Time spent T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 1.397s user 0.007s sys 0.014s
I20260808 20:14:52.777467 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:52.852141 16182 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:52.856360 16118 leader_election.cc:288] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:52.851745 16322 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:52.897293 16121 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:52.853204 13072 raft_consensus.cc:1226] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 LEADER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7. Ops: []
I20260808 20:14:52.838641 16134 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:52.777451 12241 rpcz_store.cc:269] 0808 20:14:49.822253 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:49.822413 (+ 160us) service_pool.cc:227] Handling call
0808 20:14:52.677248 (+2854835us) spinlock_profiling.cc:243] Waited 1.73 ms on lock 0x7b58001a1398. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:52.747440 (+ 70192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3975424}
W20260808 20:14:52.826303 12244 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1844) took 2869 ms (client timeout 1796 ms). Trace:
I20260808 20:14:52.756822 15873 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:52.853394 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:52.752592 16201 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:52.759626 12138 rpcz_store.cc:269] 0808 20:14:49.710045 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:49.727736 (+ 17691us) service_pool.cc:227] Handling call
0808 20:14:52.173757 (+2446021us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537ac6c 00007fa96537abd6 000000000034aea9 00007fa96d1a6a30 00007fa96d26c7b4 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:52.530688 (+356931us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":29988608}
I20260808 20:14:52.857646 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 403) took 1995 ms. Trace:
I20260808 20:14:52.954993 12142 rpcz_store.cc:276] 0808 20:14:50.862401 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:52.820914 (+1958513us) service_pool.cc:227] Handling call
0808 20:14:52.857113 (+ 36199us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:52.857302 (+ 189us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:52.857332 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:52.857376 (+ 44us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:52.857406 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:52.857468 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:52.857586 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:52.959184 12243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1838) took 3229 ms. Trace:
I20260808 20:14:52.917429 16291 leader_election.cc:288] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:52.959579 12243 rpcz_store.cc:276] 0808 20:14:49.730192 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:49.730298 (+ 106us) service_pool.cc:227] Handling call
0808 20:14:49.762426 (+ 32128us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:50.113494 (+351068us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:50.113527 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:50.130947 (+ 17420us) log.cc:842] Serialized 26 byte log entry
0808 20:14:50.775134 (+644187us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:14:51.216458 (+441324us) log.cc:842] Serialized 14 byte log entry
0808 20:14:51.244431 (+ 27973us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:52.958761 (+1714330us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b540007d854. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:52.958819 (+ 58us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:52.958871 (+ 52us) raft_consensus.cc:1671] finished
0808 20:14:52.958964 (+ 93us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:52.959124 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":131994,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":969842,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":33389,"spinlock_wait_cycles":476110080,"thread_start_us":980226,"threads_started":2,"wal-append.queue_time_us":769487}
I20260808 20:14:52.843338 12140 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
W20260808 20:14:52.972641 12140 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1765) took 1848 ms (client timeout 2327 ms). Trace:
W20260808 20:14:52.975373 12140 rpcz_store.cc:269] 0808 20:14:51.124373 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:52.757388 (+1633015us) service_pool.cc:227] Handling call
0808 20:14:52.972572 (+215184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:52.881098 16304 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:52.996287 13018 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.311s user 0.014s sys 0.047s
W20260808 20:14:52.997617 13018 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.312s user 0.015s sys 0.048s
W20260808 20:14:52.989543 12001 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.174s user 0.011s sys 0.074s
W20260808 20:14:53.003129 12001 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.187s user 0.011s sys 0.075s
W20260808 20:14:53.003489 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 1.18785e+06us
W20260808 20:14:52.977005 15666 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.783s user 0.021s sys 0.015s
I20260808 20:14:53.045670 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:53.049443 12248 raft_consensus.cc:2454] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
W20260808 20:14:53.049952 12248 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2001) took 3226 ms (client timeout 1551 ms). Trace:
W20260808 20:14:53.050307 12248 rpcz_store.cc:269] 0808 20:14:49.823421 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:49.823688 (+ 267us) service_pool.cc:227] Handling call
0808 20:14:53.045492 (+3221804us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b58001a1398. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:53.049885 (+ 4393us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25076480}
I20260808 20:14:52.843698 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:52.854440 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:52.908478 12146 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1769) took 1718 ms (client timeout 2144 ms). Trace:
W20260808 20:14:53.062072 12146 rpcz_store.cc:269] 0808 20:14:51.190173 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:52.698036 (+1507863us) service_pool.cc:227] Handling call
0808 20:14:52.908411 (+210375us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:52.908515 12147 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1766) took 1783 ms (client timeout 2124 ms). Trace:
W20260808 20:14:53.062896 12147 rpcz_store.cc:269] 0808 20:14:51.124616 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:52.699192 (+1574576us) service_pool.cc:227] Handling call
0808 20:14:52.908466 (+209274us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:52.910502 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 1.14393e+06us
W20260808 20:14:52.912402 12150 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1764) took 1788 ms (client timeout 2069 ms). Trace:
W20260808 20:14:52.901502 12000 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.131s user 0.048s sys 0.079s
I20260808 20:14:52.915776 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:52.918205 13016 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.161s user 0.046s sys 0.090s
I20260808 20:14:52.858325 16327 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:52.919073 12249 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:52.915767 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:52.949196 13078 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Advancing to term 8
W20260808 20:14:52.947893 13014 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.960s user 0.030s sys 0.038s
W20260808 20:14:52.949712 12244 rpcz_store.cc:269] 0808 20:14:49.957417 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:49.957974 (+ 557us) service_pool.cc:227] Handling call
0808 20:14:52.342487 (+2384513us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b58001a1398. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:52.778068 (+435581us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24582016}
I20260808 20:14:52.902145 16312 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:52.771476 16271 leader_election.cc:288] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:53.061569 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:52.947875 15826 consensus_queue.cc:260] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:53.134666 12144 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1464) took 3422 ms (client timeout 2070 ms). Trace:
W20260808 20:14:53.774554 13014 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.787s user 0.043s sys 0.051s
I20260808 20:14:53.631070 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:53.586695 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:53.586265 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:53.778292 12241 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
W20260808 20:14:53.134821 12101 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.928s user 0.020s sys 0.069s
W20260808 20:14:53.774559 12249 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2012) took 2909 ms (client timeout 1965 ms). Trace:
W20260808 20:14:53.706543 13074 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53290 (request call id 640) took 1908 ms (client timeout 2124 ms). Trace:
W20260808 20:14:53.775429 13016 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.018s user 0.061s sys 0.121s
I20260808 20:14:53.778352 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:53.141623 11995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.388s user 0.001s sys 0.021s
I20260808 20:14:53.148212 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:53.775571 12000 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.006s user 0.049s sys 0.111s
W20260808 20:14:53.343602 12040 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 2008) took 3878 ms (client timeout 2063 ms). Trace:
W20260808 20:14:53.780869 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 2.01149e+06us
W20260808 20:14:53.779511 12101 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.573s user 0.023s sys 0.107s
W20260808 20:14:53.780705 11995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.027s user 0.006s sys 0.029s
I20260808 20:14:53.590900 12142 raft_consensus.cc:1261] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260808 20:14:53.776727 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:53.119562 12203 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.101s user 0.006s sys 0.012s
I20260808 20:14:53.780911 12251 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
W20260808 20:14:53.783273 12203 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.765s user 0.009s sys 0.024s
W20260808 20:14:53.783443 12251 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1843) took 3919 ms (client timeout 2002 ms). Trace:
W20260808 20:14:53.781038 12040 rpcz_store.cc:269] 0808 20:14:49.341672 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:49.342229 (+ 557us) service_pool.cc:227] Handling call
0808 20:14:53.219512 (+3877283us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b54000f5d54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:53.219642 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3561672576}
W20260808 20:14:53.779567 12241 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2013) took 2893 ms (client timeout 2192 ms). Trace:
I20260808 20:14:53.221349 12141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1752) took 3396 ms. Trace:
W20260808 20:14:53.161993 12245 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2010) took 2321 ms (client timeout 2062 ms). Trace:
W20260808 20:14:53.220958 12152 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1767) took 2080 ms (client timeout 1723 ms). Trace:
W20260808 20:14:53.784741 12241 rpcz_store.cc:269] 0808 20:14:50.886080 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:52.954180 (+2068100us) service_pool.cc:227] Handling call
0808 20:14:53.779505 (+825325us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:53.781896 13074 rpcz_store.cc:269] 0808 20:14:51.797717 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:51.809508 (+ 11791us) service_pool.cc:227] Handling call
0808 20:14:53.706366 (+1896858us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b5400524ed4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:53.706479 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":879780224}
W20260808 20:14:53.786233 12245 rpcz_store.cc:269] 0808 20:14:50.841053 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:52.775892 (+1934839us) service_pool.cc:227] Handling call
0808 20:14:53.161897 (+386005us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:53.784809 12036 raft_consensus.cc:2454] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
W20260808 20:14:53.775707 12243 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53030 (request call id 513) took 1973 ms (client timeout 1819 ms). Trace:
I20260808 20:14:53.756940 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1757) took 2885 ms. Trace:
W20260808 20:14:53.774955 12150 rpcz_store.cc:269] 0808 20:14:51.124136 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:52.699403 (+1575267us) service_pool.cc:227] Handling call
0808 20:14:52.912331 (+212928us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:53.775449 12144 rpcz_store.cc:269] 0808 20:14:49.711890 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:49.737583 (+ 25693us) service_pool.cc:227] Handling call
0808 20:14:53.134471 (+3396888us) spinlock_profiling.cc:243] Waited 825 ms on lock 0x7b540022fbd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:53.134590 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1897978112}
I20260808 20:14:53.785156 12141 rpcz_store.cc:276] 0808 20:14:49.824952 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:50.266335 (+441383us) service_pool.cc:227] Handling call
0808 20:14:52.197918 (+1931583us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:52.269973 (+ 72055us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:53.221168 (+951195us) spinlock_profiling.cc:243] Waited 1.87 s on lock 0x7b5400017050. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d9a09 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:53.221292 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4298137856}
W20260808 20:14:53.788740 12248 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53030 (request call id 512) took 1987 ms (client timeout 2089 ms). Trace:
I20260808 20:14:53.474809 12250 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:44376 (request call id 706) took 3494 ms. Trace:
I20260808 20:14:53.220170 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1514) took 3584 ms. Trace:
I20260808 20:14:53.789983 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:53.785808 12152 rpcz_store.cc:269] 0808 20:14:51.140401 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:52.335483 (+1195082us) service_pool.cc:227] Handling call
0808 20:14:53.220755 (+885272us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b54002a5b54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:53.220857 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":281789312}
I20260808 20:14:53.790864 12241 raft_consensus.cc:2454] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6.
I20260808 20:14:53.751856 12137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1467) took 4038 ms. Trace:
W20260808 20:14:53.787432 12243 rpcz_store.cc:269] 0808 20:14:51.802120 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:52.991057 (+1188937us) service_pool.cc:227] Handling call
0808 20:14:53.775642 (+784585us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:53.787756 12146 rpcz_store.cc:276] 0808 20:14:50.871635 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.062759 (+2191124us) service_pool.cc:227] Handling call
0808 20:14:53.756293 (+693534us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:53.756602 (+ 309us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:53.756627 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:53.756663 (+ 36us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:53.756694 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:53.756754 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:53.756864 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:53.791503 12241 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53030 (request call id 514) took 1989 ms (client timeout 2227 ms). Trace:
W20260808 20:14:53.793712 12241 rpcz_store.cc:269] 0808 20:14:51.802484 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.786425 (+1983941us) service_pool.cc:227] Handling call
0808 20:14:53.791446 (+ 5021us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:53.793529 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:53.789386 12248 rpcz_store.cc:269] 0808 20:14:51.801735 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.082260 (+1280525us) service_pool.cc:227] Handling call
0808 20:14:53.788686 (+706426us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:53.793212 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:53.796588 12047 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
W20260808 20:14:53.790766 12251 rpcz_store.cc:269] 0808 20:14:49.864494 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:49.864625 (+ 131us) service_pool.cc:227] Handling call
0808 20:14:53.148035 (+3283410us) spinlock_profiling.cc:243] Waited 2.1 ms on lock 0x7b58001a1398. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:53.783383 (+635348us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4830464}
W20260808 20:14:53.779992 12249 rpcz_store.cc:269] 0808 20:14:50.865488 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:52.765899 (+1900411us) service_pool.cc:227] Handling call
0808 20:14:53.774488 (+1008589us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:53.039716 16269 log.cc:924] Time spent T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 3.048s user 0.004s sys 0.010s
I20260808 20:14:53.789839 12250 rpcz_store.cc:276] 0808 20:14:49.850212 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:49.850582 (+ 370us) service_pool.cc:227] Handling call
0808 20:14:53.222398 (+3371816us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b58001a1398. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c726 00007fa96d26c900 00007fa96d21fb50 00007fa966916807 00007fa966916792 00007fa9669166c1
0808 20:14:53.344989 (+122591us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24408448}
I20260808 20:14:53.827807 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:53.798166 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:53.792236 12137 rpcz_store.cc:276] 0808 20:14:49.713211 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:50.266277 (+553066us) service_pool.cc:227] Handling call
0808 20:14:50.326909 (+ 60632us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:50.334062 (+ 7153us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:50.334087 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:50.467165 (+133078us) log.cc:842] Serialized 26 byte log entry
0808 20:14:51.054705 (+587540us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:51.054743 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:51.054807 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:52.898493 (+1843686us) spinlock_profiling.cc:243] Waited 77.1 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:53.751484 (+852991us) raft_consensus.cc:1671] finished
0808 20:14:53.751507 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:53.751792 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9619,"raft.queue_time_us":22423,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":123593,"spinlock_wait_cycles":177435008,"thread_start_us":425235,"threads_started":1,"wal-append.queue_time_us":731415}
I20260808 20:14:53.827596 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:53.828851 12241 raft_consensus.cc:2396] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260808 20:14:53.829556 12244 raft_consensus.cc:2454] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:53.829594 12251 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:40912
W20260808 20:14:53.060153 16321 consensus_meta.cc:287] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.767s user 0.010s sys 0.004s
I20260808 20:14:53.831426 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:53.785279 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 404) took 2922 ms. Trace:
I20260808 20:14:53.833361 12144 raft_consensus.cc:1226] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 10 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 9. Current term is 10. Ops: []
W20260808 20:14:53.832531 12244 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2016) took 1470 ms (client timeout 1662 ms). Trace:
I20260808 20:14:53.836865 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:53.833847 12241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2015) took 1472 ms. Trace:
W20260808 20:14:53.837287 12244 rpcz_store.cc:269] 0808 20:14:52.362127 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.797896 (+1435769us) service_pool.cc:227] Handling call
0808 20:14:53.832467 (+ 34571us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:53.837803 12241 rpcz_store.cc:276] 0808 20:14:52.361837 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.798745 (+1436908us) service_pool.cc:227] Handling call
0808 20:14:53.833791 (+ 35046us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:53.839164 12244 raft_consensus.cc:1261] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260808 20:14:53.839074 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:53.837488 12144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1759) took 2965 ms. Trace:
I20260808 20:14:53.840417 12144 rpcz_store.cc:276] 0808 20:14:50.872132 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.827478 (+2955346us) service_pool.cc:227] Handling call
0808 20:14:53.833187 (+ 5709us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:53.837266 (+ 4079us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:53.837431 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:53.562189 16194 consensus_meta.cc:287] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 1.301s user 0.010s sys 0.002s
W20260808 20:14:53.834792 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 243558us
W20260808 20:14:53.432457 16231 log.cc:924] Time spent T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 4.153s user 0.006s sys 0.022s
I20260808 20:14:53.837139 12142 rpcz_store.cc:276] 0808 20:14:50.862683 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:52.963020 (+2100337us) service_pool.cc:227] Handling call
0808 20:14:53.590569 (+627549us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:53.785033 (+194464us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:53.785225 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:53.845981 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:53.066614 15666 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.872s user 0.022s sys 0.015s
I20260808 20:14:53.836793 12245 raft_consensus.cc:2454] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:53.852353 12245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2017) took 1488 ms. Trace:
I20260808 20:14:53.852734 12245 rpcz_store.cc:276] 0808 20:14:52.364311 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.802537 (+1438226us) service_pool.cc:227] Handling call
0808 20:14:53.852294 (+ 49757us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:53.690884 16324 log.cc:924] Time spent T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 2.209s user 0.004s sys 0.014s
W20260808 20:14:53.123147 16189 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.026s user 0.000s sys 0.014s
I20260808 20:14:53.379784 16201 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:53.733489 16359 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:53.429203 15892 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:53.529488 12138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 405) took 2666 ms. Trace:
I20260808 20:14:53.837502 12036 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 7.
I20260808 20:14:53.587070 16173 raft_consensus.cc:3037] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:53.843489 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:53.839694 12244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1847) took 3776 ms. Trace:
I20260808 20:14:53.587255 16399 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:53.695356 16306 consensus_meta.cc:287] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.768s user 0.009s sys 0.000s
I20260808 20:14:53.849954 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:53.124464 16163 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:53.586901 16134 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:53.757118 16005 raft_consensus.cc:3037] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:53.621363 16008 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:14:53.831449 12000 connection.cc:232] RPC call timeout handler was delayed by -2.1379647490102798s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21520000000000003s for a response
I20260808 20:14:53.856489 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:53.752038 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:53.380012 16265 leader_election.cc:288] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:53.427838 16355 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:53.866050 12247 raft_consensus.cc:2454] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:14:53.798169 16266 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:53.832965 16271 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:53.111300 16396 consensus_queue.cc:1046] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.859s
I20260808 20:14:53.874572 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:54.821199 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:54.822935 12244 rpcz_store.cc:276] 0808 20:14:50.063279 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.838620 (+3775341us) service_pool.cc:227] Handling call
0808 20:14:53.838881 (+ 261us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:53.839491 (+ 610us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:53.839647 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:54.823493 12137 raft_consensus.cc:2379] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 4: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:54.823931 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1523) took 1032 ms. Trace:
I20260808 20:14:54.824254 12036 rpcz_store.cc:276] 0808 20:14:53.791994 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.827365 (+ 35371us) service_pool.cc:227] Handling call
0808 20:14:54.823877 (+996512us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:54.824608 12150 raft_consensus.cc:2454] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:14:54.825176 16266 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:54.826172 12251 raft_consensus.cc:595] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:54.826601 12251 raft_consensus.cc:603] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 20:14:54.827124 12251 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:40912 (request call id 24) took 2136 ms. Trace:
I20260808 20:14:54.827334 16266 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:14:54.511243 13078 consensus_meta.cc:287] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.736s user 0.007s sys 0.000s
I20260808 20:14:53.842482 16321 raft_consensus.cc:504] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:54.477859 16263 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:54.828884 13071 raft_consensus.cc:2354] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6.
W20260808 20:14:54.056463 13080 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.776s user 0.040s sys 0.111s
W20260808 20:14:54.829741 13080 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.549s user 0.058s sys 0.157s
I20260808 20:14:54.829852 16321 leader_election.cc:288] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:53.834160 16327 raft_consensus.cc:482] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:54.829059 16306 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:53.922195 16286 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:54.831681 12146 raft_consensus.cc:2396] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260808 20:14:54.831569 16320 raft_consensus.cc:2726] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:54.852495 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1475) took 1004 ms. Trace:
I20260808 20:14:54.852826 12146 rpcz_store.cc:276] 0808 20:14:53.848367 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.849736 (+ 1369us) service_pool.cc:227] Handling call
0808 20:14:54.852439 (+1002703us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:54.854473 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:54.828037 13078 raft_consensus.cc:2454] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 8.
W20260808 20:14:54.855886 13078 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57330 (request call id 510) took 2016 ms (client timeout 1720 ms). Trace:
W20260808 20:14:54.856243 13078 rpcz_store.cc:269] 0808 20:14:52.839006 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:52.839223 (+ 217us) service_pool.cc:227] Handling call
0808 20:14:54.855815 (+2016592us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:54.832590 16396 raft_consensus.cc:3032] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Stepping down as leader of term 6
W20260808 20:14:54.481618 16406 log.cc:924] Time spent T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.365s user 0.000s sys 0.000s
I20260808 20:14:54.478116 16334 consensus_queue.cc:1035] T 1d7d7a85e3964f3f9d5f2dc39e23340b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.792s
W20260808 20:14:54.054840 12206 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.651s user 0.014s sys 0.094s
W20260808 20:14:54.900106 12206 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.496s user 0.034s sys 0.138s
W20260808 20:14:54.536769 16324 log.cc:924] Time spent T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.222s user 0.000s sys 0.001s
I20260808 20:14:54.837103 16399 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:54.697964 16349 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.362s user 0.009s sys 0.043s
I20260808 20:14:54.837713 15892 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:54.838449 12243 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 7.
W20260808 20:14:54.907542 12243 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53030 (request call id 510) took 3106 ms (client timeout 2879 ms). Trace:
W20260808 20:14:54.907928 12243 rpcz_store.cc:269] 0808 20:14:51.801091 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.793856 (+1992765us) service_pool.cc:227] Handling call
0808 20:14:54.907466 (+1113610us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39296}
I20260808 20:14:54.838973 16271 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:54.838769 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:54.909121 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:53.795872 12040 raft_consensus.cc:2454] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:54.909840 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1524) took 1117 ms. Trace:
I20260808 20:14:54.910156 12040 rpcz_store.cc:276] 0808 20:14:53.792383 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.793289 (+ 906us) service_pool.cc:227] Handling call
0808 20:14:54.909778 (+1116489us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:54.153892 16292 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:54.514341 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:54.823664 13072 raft_consensus.cc:2379] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 5: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:54.821523 15826 raft_consensus.cc:3037] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:54.092734 12046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 565) took 5373 ms. Trace:
I20260808 20:14:53.916307 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:54.842334 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:54.918712 12137 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:54.904215 16173 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:54.843947 15592 raft_consensus.cc:482] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:54.904234 16134 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:54.026499 15666 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
W20260808 20:14:53.941612 16357 consensus_meta.cc:287] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 1.176s user 0.007s sys 0.007s
I20260808 20:14:53.789850 12041 rpcz_store.cc:276] 0808 20:14:49.635570 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:49.635739 (+ 169us) service_pool.cc:227] Handling call
0808 20:14:49.636020 (+ 281us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:49.644265 (+ 8245us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:49.644297 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:49.644340 (+ 43us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:49.644370 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:53.219943 (+3575573us) spinlock_profiling.cc:243] Waited 8.04 ms on lock 0x7b540038ccd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:53.219990 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:53.220112 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18496768}
I20260808 20:14:54.925565 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:53.866947 16189 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.853s user 0.004s sys 0.027s
W20260808 20:14:53.970942 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 195701us
I20260808 20:14:54.929004 13015 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:54.846197 12138 rpcz_store.cc:276] 0808 20:14:50.862914 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:52.963545 (+2100631us) service_pool.cc:227] Handling call
0808 20:14:53.125282 (+161737us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:53.192136 (+ 66854us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:53.213871 (+ 21735us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:53.241668 (+ 27797us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:53.374902 (+133234us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:53.529149 (+154247us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:53.529390 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:14:54.169461 12207 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.449s user 0.012s sys 0.035s
W20260808 20:14:53.904210 12101 connection.cc:232] RPC call timeout handler was delayed by -2.5650308220647275s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15510000000000002s for a response
I20260808 20:14:53.843956 12250 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 508) took 3739 ms. Trace:
W20260808 20:14:54.937793 12247 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2011) took 4079 ms (client timeout 1908 ms). Trace:
W20260808 20:14:54.169140 12048 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.819s user 0.010s sys 0.033s
W20260808 20:14:54.168973 13079 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.263s user 0.000s sys 0.017s
I20260808 20:14:53.867558 12249 raft_consensus.cc:1261] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260808 20:14:54.938202 12247 rpcz_store.cc:269] 0808 20:14:50.857922 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:51.759522 (+901600us) service_pool.cc:227] Handling call
0808 20:14:54.937626 (+3178104us) spinlock_profiling.cc:243] Waited 2.11 s on lock 0x7b540029d954. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:54.937727 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4842642176}
I20260808 20:14:54.938699 12249 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1846) took 4900 ms. Trace:
I20260808 20:14:54.939033 12249 rpcz_store.cc:276] 0808 20:14:50.038149 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.841851 (+3803702us) service_pool.cc:227] Handling call
0808 20:14:53.867276 (+ 25425us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:54.938433 (+1071157us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:54.938640 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:53.874104 12141 raft_consensus.cc:1261] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 20:14:54.831439 16008 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:54.822031 12245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 509) took 4716 ms. Trace:
I20260808 20:14:54.866519 12146 raft_consensus.cc:2379] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 8: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:54.946813 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:54.906757 16201 raft_consensus.cc:482] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:54.857985 12251 rpcz_store.cc:276] 0808 20:14:52.691204 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.828348 (+1137144us) service_pool.cc:227] Handling call
0808 20:14:54.827038 (+998690us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:54.946008 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:54.948072 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1530) took 1043 ms. Trace:
W20260808 20:14:54.889909 12105 connection.cc:232] RPC call timeout handler was delayed by -2.118538660928607s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22460000000000002s for a response
I20260808 20:14:54.899170 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:54.056195 16372 consensus_meta.cc:287] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 1.298s user 0.015s sys 0.000s
I20260808 20:14:53.947847 15699 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:54.906005 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:54.909441 12248 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2014) took 3979 ms. Trace:
I20260808 20:14:54.950016 12141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1474) took 1103 ms. Trace:
I20260808 20:14:54.911125 12142 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 7.
I20260808 20:14:54.867271 16396 raft_consensus.cc:729] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:54.917932 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:54.951030 12251 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2007) took 4164 ms. Trace:
I20260808 20:14:54.954159 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:54.922684 12047 raft_consensus.cc:1261] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260808 20:14:54.907083 16349 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.575s user 0.013s sys 0.050s
I20260808 20:14:54.897816 16265 raft_consensus.cc:482] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:54.961866 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:54.920526 16173 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:54.921178 15592 raft_consensus.cc:504] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:54.918283 12152 raft_consensus.cc:2454] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:14:54.968712 12247 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2006) took 4206 ms. Trace:
W20260808 20:14:54.928067 13072 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57330 (request call id 514) took 1976 ms (client timeout 1884 ms). Trace:
I20260808 20:14:54.970889 12254 raft_consensus.cc:2454] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
W20260808 20:14:54.971127 13072 rpcz_store.cc:269] 0808 20:14:52.951743 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.751496 (+799753us) service_pool.cc:227] Handling call
0808 20:14:54.927984 (+1176488us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:54.972097 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:54.972711 13072 raft_consensus.cc:2454] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:54.973766 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:54.940542 12101 connection.cc:232] RPC call timeout handler was delayed by -2.3598308218643069s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1779s for a response
W20260808 20:14:54.940845 12048 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.591s user 0.017s sys 0.039s
W20260808 20:14:54.941092 12207 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 2.221s user 0.016s sys 0.044s
I20260808 20:14:54.937664 15666 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:54.976285 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 2.28286e+06us
I20260808 20:14:54.962100 15699 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:54.977558 15666 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:54.979012 15699 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:54.980592 12207 connection.cc:232] RPC call timeout handler was delayed by -2.2596332640387118s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17200000000000001s for a response
W20260808 20:14:54.983213 12001 connection.cc:232] RPC call timeout handler was delayed by -1.941777670988813s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23270000000000002s for a response
I20260808 20:14:54.965790 16349 raft_consensus.cc:482] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:54.986712 16349 raft_consensus.cc:504] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:54.938099 12250 rpcz_store.cc:276] 0808 20:14:50.104812 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.843102 (+3738290us) service_pool.cc:227] Handling call
0808 20:14:53.843519 (+ 417us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:53.843685 (+ 166us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:53.843708 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:53.843743 (+ 35us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:53.843770 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:53.843818 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:53.843906 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39040}
I20260808 20:14:54.926127 12041 raft_consensus.cc:2454] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:54.947028 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:54.830490 16163 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:54.955657 13014 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:54.945976 12245 rpcz_store.cc:276] 0808 20:14:50.105373 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:54.821152 (+4715779us) service_pool.cc:227] Handling call
0808 20:14:54.821479 (+ 327us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:54.821668 (+ 189us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:54.821692 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:54.821745 (+ 53us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:54.821774 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:54.821828 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:54.821969 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:54.992164 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:54.995316 12250 raft_consensus.cc:2454] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:54.992430 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:54.993287 13078 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 7.
I20260808 20:14:54.830564 16327 raft_consensus.cc:504] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:54.957803 13075 raft_consensus.cc:2454] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:54.957268 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1472) took 1111 ms. Trace:
I20260808 20:14:54.959018 12137 raft_consensus.cc:2454] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
I20260808 20:14:54.957355 12141 rpcz_store.cc:276] 0808 20:14:53.846749 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.846865 (+ 116us) service_pool.cc:227] Handling call
0808 20:14:53.873791 (+ 26926us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:54.939803 (+1066012us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:54.949938 (+ 10135us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:54.957496 12150 raft_consensus.cc:2454] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:54.956806 12040 raft_consensus.cc:2379] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 8: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:54.962538 12146 raft_consensus.cc:2454] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:54.962423 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:54.916625 12046 rpcz_store.cc:276] 0808 20:14:48.692027 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:48.692148 (+ 121us) service_pool.cc:227] Handling call
0808 20:14:48.715304 (+ 23156us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:48.721033 (+ 5729us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:48.721066 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:48.749987 (+ 28921us) log.cc:842] Serialized 26 byte log entry
0808 20:14:49.639446 (+889459us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:49.639485 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:49.639541 (+ 56us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:52.901102 (+3261561us) spinlock_profiling.cc:243] Waited 9.77 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:54.055691 (+1154589us) raft_consensus.cc:1671] finished
0808 20:14:54.055714 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:54.064922 (+ 9208us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12808,"raft.queue_time_us":5599,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":7790,"spinlock_wait_cycles":22463744,"thread_start_us":822789,"threads_started":1,"wal-append.queue_time_us":991217}
I20260808 20:14:54.956147 12248 rpcz_store.cc:276] 0808 20:14:50.930403 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.831861 (+2901458us) service_pool.cc:227] Handling call
0808 20:14:54.909383 (+1077522us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:54.965497 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:54.948380 12036 rpcz_store.cc:276] 0808 20:14:53.905084 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:54.939182 (+1034098us) service_pool.cc:227] Handling call
0808 20:14:54.948017 (+ 8835us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:54.956606 12251 rpcz_store.cc:276] 0808 20:14:50.786808 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:54.950235 (+4163427us) service_pool.cc:227] Handling call
0808 20:14:54.950554 (+ 319us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:54.950736 (+ 182us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:54.950762 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:54.950801 (+ 39us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:14:54.950834 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:54.950880 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:54.950971 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:54.962855 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:54.967927 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:54.920850 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:54.968679 12152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1471) took 1122 ms. Trace:
W20260808 20:14:54.941332 13079 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.035s user 0.000s sys 0.030s
I20260808 20:14:54.969125 12247 rpcz_store.cc:276] 0808 20:14:50.762200 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:54.949033 (+4186833us) service_pool.cc:227] Handling call
0808 20:14:54.968206 (+ 19173us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:54.968398 (+ 192us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:54.968425 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:54.968476 (+ 51us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:54.968511 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:54.968558 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:54.968656 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:54.990011 13076 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:14:54.908317 16263 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:54.888474 16286 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:54.949522 16189 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:55.063202 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:54.924609 16134 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:55.056376 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:54.988447 11894 catalog_manager.cc:5495] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } }
W20260808 20:14:55.083850 12039 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1529) took 1216 ms (client timeout 1529 ms). Trace:
I20260808 20:14:55.057603 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:55.058420 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:54.916862 16292 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:55.055910 13073 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 7.
I20260808 20:14:55.059563 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.061069 12249 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 7.
I20260808 20:14:55.101815 12041 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 7.
I20260808 20:14:55.102537 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:55.061306 12142 rpcz_store.cc:276] 0808 20:14:53.846191 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.917887 (+ 71696us) service_pool.cc:227] Handling call
0808 20:14:54.957212 (+1039325us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:55.062100 12047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 2010) took 1224 ms. Trace:
I20260808 20:14:55.055369 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 2013) took 1187 ms. Trace:
I20260808 20:14:55.059197 12152 rpcz_store.cc:276] 0808 20:14:53.845922 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.847152 (+ 1230us) service_pool.cc:227] Handling call
0808 20:14:54.968622 (+1121470us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:55.069438 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:55.103236 12047 rpcz_store.cc:276] 0808 20:14:53.831222 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.831406 (+ 184us) service_pool.cc:227] Handling call
0808 20:14:53.831692 (+ 286us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:54.958092 (+1126400us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:55.055063 (+ 96971us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540046ffd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:55.055244 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2508290944}
I20260808 20:14:55.103848 12146 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:14:55.070197 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:55.104511 12205 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:55.104880 12137 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Advancing to term 10
I20260808 20:14:55.073251 13015 leader_election.cc:302] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:55.083573 11894 catalog_manager.cc:5495] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 1 to 5, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4). New cstate: current_term: 5 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:55.076570 16286 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:55.078593 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:55.093310 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 137717us
I20260808 20:14:55.106843 13075 raft_consensus.cc:2454] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:55.106873 13078 raft_consensus.cc:2454] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:55.107558 12040 rpcz_store.cc:276] 0808 20:14:53.867715 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:54.917696 (+1049981us) service_pool.cc:227] Handling call
0808 20:14:55.055305 (+137609us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:55.097319 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:55.108325 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.110111 12038 raft_consensus.cc:2454] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:55.108939 12248 raft_consensus.cc:2454] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:14:55.112008 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:55.112942 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:55.113555 12247 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 9.
I20260808 20:14:55.083567 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.118431 12036 raft_consensus.cc:2454] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 2.
I20260808 20:14:55.070945 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:54.992615 16306 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 9 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:54.988826 16349 leader_election.cc:288] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:54.937558 16357 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:55.100515 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:55.102628 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:54.960400 16372 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:55.107379 13014 leader_election.cc:302] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:55.149513 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:55.125834 12041 raft_consensus.cc:2454] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
W20260808 20:14:55.084337 12039 rpcz_store.cc:269] 0808 20:14:53.867879 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:54.920602 (+1052723us) service_pool.cc:227] Handling call
0808 20:14:55.083792 (+163190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:55.121104 12249 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 7.
I20260808 20:14:55.150906 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:54.989091 16359 raft_consensus.cc:2781] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:14:54.963250 16407 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:55.117738 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:55.064179 16173 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:54.989490 13071 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 740) took 5275 ms (client timeout 2246 ms). Trace:
I20260808 20:14:54.991098 16399 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:55.060737 16178 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:55.098645 15931 raft_consensus.cc:2781] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:54.969316 16265 raft_consensus.cc:504] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:55.173882 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:55.128667 11894 catalog_manager.cc:5495] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 2 to 4, leader changed from e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 4 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:55.175802 11894 catalog_manager.cc:5495] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 5 to 6, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 6 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } }
W20260808 20:14:55.177086 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 62668.9us
I20260808 20:14:55.119674 12150 raft_consensus.cc:2379] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:55.105377 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:55.126742 15666 raft_consensus.cc:2781] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:55.146592 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.108982 12047 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:14:55.189468 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.152580 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:55.153453 12141 raft_consensus.cc:2454] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:55.108379 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.190510 12248 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:54.955112 16201 raft_consensus.cc:504] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:55.154767 12243 raft_consensus.cc:1261] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260808 20:14:55.154181 16359 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:55.166965 16005 raft_consensus.cc:2726] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:55.149009 16306 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:55.208120 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.099984 16189 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:55.160192 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.181094 16173 raft_consensus.cc:482] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:55.161275 12249 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2008) took 4354 ms. Trace:
I20260808 20:14:55.242308 12249 rpcz_store.cc:276] 0808 20:14:50.807347 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:55.151417 (+4344070us) service_pool.cc:227] Handling call
0808 20:14:55.160809 (+ 9392us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:55.161002 (+ 193us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:55.161026 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:55.161064 (+ 38us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:55.161094 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:55.161140 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:55.161228 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:55.244046 12249 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 511) took 3442 ms. Trace:
I20260808 20:14:55.244378 12249 rpcz_store.cc:276] 0808 20:14:51.801401 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:55.243333 (+3441932us) service_pool.cc:227] Handling call
0808 20:14:55.243619 (+ 286us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:55.243761 (+ 142us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:55.243781 (+ 20us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:55.243815 (+ 34us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:55.243843 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:55.243887 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:55.243994 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:55.183722 16399 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
W20260808 20:14:55.161470 12254 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1849) took 4992 ms (client timeout 1753 ms). Trace:
W20260808 20:14:55.162669 13076 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53290 (request call id 634) took 5060 ms (client timeout 1822 ms). Trace:
I20260808 20:14:55.165444 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:55.166126 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:55.127566 12251 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "21889249378248049ef043f0588cebc9"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:40912
W20260808 20:14:55.258704 13076 rpcz_store.cc:269] 0808 20:14:50.102726 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:50.127372 (+ 24646us) service_pool.cc:227] Handling call
0808 20:14:55.162470 (+5035098us) spinlock_profiling.cc:243] Waited 4.73 s on lock 0x7b5400581454. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:55.162581 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10884595968}
I20260808 20:14:55.259044 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:55.259626 12150 raft_consensus.cc:2454] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:14:55.259976 13073 raft_consensus.cc:2396] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260808 20:14:55.260437 12251 raft_consensus.cc:595] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:55.260798 12251 raft_consensus.cc:603] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 20:14:55.170078 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:55.185276 12043 raft_consensus.cc:2379] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 7: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:55.187127 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.262427 12041 raft_consensus.cc:2379] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 5: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:55.192730 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.263036 12040 raft_consensus.cc:2379] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 7: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:55.155882 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:55.194041 12046 raft_consensus.cc:2396] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 8 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260808 20:14:55.110682 15653 consensus_queue.cc:1035] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.074s
W20260808 20:14:55.198946 11994 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 2.426s user 0.082s sys 0.142s
W20260808 20:14:55.266242 11994 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 2.493s user 0.083s sys 0.143s
W20260808 20:14:55.266558 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 2.49403e+06us
I20260808 20:14:55.181697 15931 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:55.188325 15666 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:55.276947 13076 raft_consensus.cc:1261] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 20:14:55.220485 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.159010 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:55.278661 12152 raft_consensus.cc:3037] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 6
I20260808 20:14:55.273808 15931 raft_consensus.cc:3037] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:55.282231 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.283126 12047 raft_consensus.cc:2396] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260808 20:14:55.124032 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:55.285789 12036 raft_consensus.cc:3037] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:14:55.260459 12254 rpcz_store.cc:269] 0808 20:14:50.168884 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:50.536621 (+367737us) service_pool.cc:227] Handling call
0808 20:14:55.161283 (+4624662us) spinlock_profiling.cc:243] Waited 4.43 s on lock 0x7b5400165654. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:55.161375 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10195852032}
I20260808 20:14:55.260809 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:55.228927 11894 catalog_manager.cc:5495] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 2 to 5, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 5 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } }
I20260808 20:14:55.289830 11894 catalog_manager.cc:5495] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:55.291824 11894 catalog_manager.cc:5495] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:14:55.230278 13016 leader_election.cc:302] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:55.240347 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:55.199612 16359 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:55.158362 13014 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:55.294824 12039 raft_consensus.cc:1261] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260808 20:14:55.294977 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 136690us
I20260808 20:14:55.214262 16286 raft_consensus.cc:482] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:55.295595 13014 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:55.269793 12249 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2018) took 2897 ms. Trace:
W20260808 20:14:55.270257 12045 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.647s user 0.027s sys 0.121s
W20260808 20:14:55.205029 16371 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.351s user 0.030s sys 0.085s
I20260808 20:14:55.123078 16396 consensus_queue.cc:260] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:55.293768 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 63591.8us
W20260808 20:14:55.296331 12045 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.673s user 0.027s sys 0.121s
I20260808 20:14:55.295382 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1527) took 1449 ms. Trace:
I20260808 20:14:55.371948 12039 rpcz_store.cc:276] 0808 20:14:53.846217 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:55.264702 (+1418485us) service_pool.cc:227] Handling call
0808 20:14:55.294549 (+ 29847us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:55.295179 (+ 630us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:55.295338 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:55.372057 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:55.296203 12249 rpcz_store.cc:276] 0808 20:14:52.372863 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:55.245221 (+2872358us) service_pool.cc:227] Handling call
0808 20:14:55.269346 (+ 24125us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:55.269512 (+ 166us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:55.269537 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:55.269574 (+ 37us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:55.269600 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:55.269648 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:55.269743 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:55.373780 12142 raft_consensus.cc:2396] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 20:14:55.379607 12247 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2019) took 2839 ms. Trace:
I20260808 20:14:55.379993 12247 rpcz_store.cc:276] 0808 20:14:52.540686 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:55.272563 (+2731877us) service_pool.cc:227] Handling call
0808 20:14:55.379121 (+106558us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:55.379286 (+ 165us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:55.379312 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:55.379350 (+ 38us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:14:55.379377 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:55.379424 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:55.379550 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:55.371209 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.171659 16407 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:55.166608 13071 rpcz_store.cc:269] 0808 20:14:49.713987 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:49.721146 (+ 7159us) service_pool.cc:227] Handling call
0808 20:14:52.733296 (+3012150us) spinlock_profiling.cc:243] Waited 46 ms on lock 0x7b5800005198. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537abce 000000000034aea9 00007fa96d1a6a30 00007fa96d26c7b4 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:14:54.989252 (+2255956us) spinlock_profiling.cc:243] Waited 2.05 s on lock 0x7b5400670554. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:54.989389 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4823155328}
I20260808 20:14:55.385847 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:55.382987 12041 raft_consensus.cc:2454] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:55.390234 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:55.403158 13078 raft_consensus.cc:2379] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 5: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:55.391085 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:55.397526 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 413810us
W20260808 20:14:55.404444 12001 connection.cc:232] RPC call timeout handler was delayed by -1.5395882150623947s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21440000000000003s for a response
I20260808 20:14:55.400024 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:55.288825 12150 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
W20260808 20:14:55.370492 16371 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.517s user 0.031s sys 0.085s
I20260808 20:14:55.304498 12248 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:44376 (request call id 707) took 4368 ms. Trace:
W20260808 20:14:55.310197 12102 leader_election.cc:334] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Timed out: RequestConsensusVote RPC to 127.10.24.1:40747 timed out after 1.551s (SENT)
W20260808 20:14:55.408118 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 97960.1us
I20260808 20:14:55.409974 13016 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:55.183910 16178 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:55.404760 12001 connection.cc:232] RPC call timeout handler was delayed by -1.2801968138664961s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
W20260808 20:14:55.406450 12144 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.557s user 0.020s sys 0.052s
W20260808 20:14:55.417874 12144 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.569s user 0.020s sys 0.052s
I20260808 20:14:55.418080 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:55.418710 13076 raft_consensus.cc:2454] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
W20260808 20:14:55.417690 12001 connection.cc:232] RPC call timeout handler was delayed by -1.1979461358860135s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22000000000000003s for a response
W20260808 20:14:55.401173 12140 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.375s user 0.056s sys 0.125s
I20260808 20:14:55.413157 12254 raft_consensus.cc:3037] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:55.407455 12248 rpcz_store.cc:276] 0808 20:14:50.907513 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:55.191069 (+4283556us) service_pool.cc:227] Handling call
0808 20:14:55.275692 (+ 84623us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:14:55.415135 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.242542 16306 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:55.421320 12041 raft_consensus.cc:2454] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:55.257424 16173 raft_consensus.cc:504] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:55.421595 12140 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.396s user 0.057s sys 0.126s
I20260808 20:14:55.383816 16388 raft_consensus.cc:2781] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 20:14:55.384356 16163 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:14:55.432407 12038 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1528) took 1582 ms (client timeout 1513 ms). Trace:
I20260808 20:14:55.429039 15908 raft_consensus.cc:2726] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:55.411499 11894 catalog_manager.cc:5495] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: config changed from index 2 to 3, term changed from 1 to 3, e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:55.440622 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:55.454486 12248 raft_consensus.cc:3032] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Stepping down as leader of term 6
W20260808 20:14:55.442407 12038 rpcz_store.cc:269] 0808 20:14:53.849801 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.923145 (+ 73344us) service_pool.cc:227] Handling call
0808 20:14:55.432228 (+1509083us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b540009ef54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:55.432333 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":369162496}
I20260808 20:14:55.454437 11894 catalog_manager.cc:5495] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
W20260808 20:14:55.442823 12241 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.398s user 0.027s sys 0.050s
I20260808 20:14:55.450843 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:55.417099 16396 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:55.474524 12254 raft_consensus.cc:1261] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260808 20:14:55.451489 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:55.481662 13073 raft_consensus.cc:2379] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:55.456647 13078 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:14:55.438797 16371 raft_consensus.cc:2781] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:14:55.481674 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.483812 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.484207 12047 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:55.484517 12038 raft_consensus.cc:2379] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 7: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:55.484987 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.485426 13076 raft_consensus.cc:1261] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260808 20:14:55.457052 12241 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.413s user 0.027s sys 0.050s
I20260808 20:14:55.298447 16312 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:55.282972 15592 leader_election.cc:288] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:55.491109 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:55.282051 15666 raft_consensus.cc:3037] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:55.454869 12248 raft_consensus.cc:729] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:55.497717 12248 consensus_queue.cc:260] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:55.498714 12248 raft_consensus.cc:3037] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:55.440062 12142 raft_consensus.cc:1261] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 20:14:55.504040 12043 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1522) took 1726 ms. Trace:
W20260808 20:14:55.458796 12102 leader_election.cc:334] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Timed out: RequestConsensusVote RPC to 127.10.24.1:40747 timed out after 1.589s (SENT)
I20260808 20:14:55.505125 12102 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:55.505512 12243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1852) took 5168 ms. Trace:
I20260808 20:14:55.459597 12137 raft_consensus.cc:2454] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 10.
I20260808 20:14:55.406800 12150 raft_consensus.cc:1261] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260808 20:14:55.505888 12243 rpcz_store.cc:276] 0808 20:14:50.337106 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:54.910126 (+4573020us) service_pool.cc:227] Handling call
0808 20:14:54.928695 (+ 18569us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:55.193264 (+264569us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:55.505286 (+312022us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b5400144454. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:55.505452 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":503269120}
W20260808 20:14:55.505978 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 47217.1us
I20260808 20:14:55.508013 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:55.508612 12137 raft_consensus.cc:2454] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:14:55.509294 16178 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:55.511657 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:55.512364 12146 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 7.
I20260808 20:14:55.518031 13016 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:55.519451 13016 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:55.519451 12248 raft_consensus.cc:1261] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260808 20:14:55.520437 12248 pending_rounds.cc:76] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a: Aborting all ops after (but not including) 6
I20260808 20:14:55.437696 16388 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:55.383376 16286 raft_consensus.cc:504] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:55.526553 13016 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:55.297353 15892 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:55.424008 16008 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:55.423821 16271 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:55.477267 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:55.472626 11894 catalog_manager.cc:5495] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 2 to 4, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 4 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } }
I20260808 20:14:55.478950 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.479218 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.479399 16407 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:55.479931 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.480713 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.538779 12040 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 7.
I20260808 20:14:55.488214 16134 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:55.485992 12247 raft_consensus.cc:1261] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260808 20:14:55.476708 15826 consensus_meta.cc:287] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.552s user 0.005s sys 0.000s
I20260808 20:14:55.490280 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.464782 12036 raft_consensus.cc:2454] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:55.469841 16173 leader_election.cc:288] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:55.520769 12248 pending_rounds.cc:97] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260808 20:14:55.550247 12248 raft_consensus.cc:2866] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:55.504421 12043 rpcz_store.cc:276] 0808 20:14:53.778020 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.778129 (+ 109us) service_pool.cc:227] Handling call
0808 20:14:55.503822 (+1725693us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54000f6254. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:55.503980 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3230236544}
I20260808 20:14:55.265776 16399 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:55.516065 12102 leader_election.cc:334] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Timed out: RequestConsensusVote RPC to 127.10.24.1:40747 timed out after 1.662s (SENT)
I20260808 20:14:55.487562 16371 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:55.553099 16371 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:55.456226 16263 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:55.529543 16321 consensus_queue.cc:1035] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 20:14:55.490005 15908 consensus_queue.cc:1035] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260808 20:14:55.543511 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:55.538729 12041 raft_consensus.cc:2454] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:14:55.537403 12039 raft_consensus.cc:2454] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:55.490636 16327 leader_election.cc:288] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:55.524149 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:55.560340 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:55.560874 12146 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6.
I20260808 20:14:55.539121 16357 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:55.552042 12047 raft_consensus.cc:2379] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 7: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:55.546437 11894 catalog_manager.cc:5495] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 3 to 4, leader changed from e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 4 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 20:14:55.517115 16292 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:55.532013 16372 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:55.530501 16388 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 FOLLOWER]: Advancing to term 10
I20260808 20:14:55.535346 16400 raft_consensus.cc:2781] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 20:14:55.543152 16178 raft_consensus.cc:2781] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:55.548481 15674 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:55.564150 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:55.578526 15674 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:55.579157 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:55.551390 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:55.548014 15892 raft_consensus.cc:2781] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:14:55.580097 12146 raft_consensus.cc:2396] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 20:14:55.565681 15826 raft_consensus.cc:2726] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:55.567175 13016 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:55.564819 12137 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6.
I20260808 20:14:55.553903 16407 consensus_queue.cc:1035] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
W20260808 20:14:55.558048 16341 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260808 20:14:55.575541 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.580099 16189 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:55.598255 15908 raft_consensus.cc:2781] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:55.605465 15908 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:55.605810 15908 raft_consensus.cc:3037] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:55.606309 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.606950 12038 raft_consensus.cc:2454] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
W20260808 20:14:55.588961 12102 leader_election.cc:334] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Timed out: RequestConsensusVote RPC to 127.10.24.1:40747 timed out after 1.779s (SENT)
I20260808 20:14:55.567934 16201 leader_election.cc:288] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:55.554747 16265 leader_election.cc:288] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:55.594630 13080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53296 (request call id 738) took 5953 ms. Trace:
I20260808 20:14:55.565572 16263 consensus_queue.cc:1035] T e256c29dec964f49856cf204f950793b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260808 20:14:55.614926 13080 rpcz_store.cc:276] 0808 20:14:49.636744 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:49.639649 (+ 2905us) service_pool.cc:227] Handling call
0808 20:14:49.639961 (+ 312us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:49.640253 (+ 292us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:49.640281 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:52.221497 (+2581216us) spinlock_profiling.cc:243] Waited 26.9 ms on lock 0x7b2c00091d88. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa966cbde11 00007fa966ccd442 00007fa968315c11 00007fa9682da58a 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772
0808 20:14:52.246609 (+ 25112us) log.cc:842] Serialized 26 byte log entry
0808 20:14:54.831338 (+2584729us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:54.831379 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:54.831477 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:55.590073 (+758596us) raft_consensus.cc:1671] finished
0808 20:14:55.590096 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:55.590294 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":24792,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":138,"spinlock_wait_cycles":61909248,"thread_start_us":2550925,"threads_started":1,"wal-append.queue_time_us":3329311}
W20260808 20:14:55.610702 12102 leader_election.cc:334] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Timed out: RequestConsensusVote RPC to 127.10.24.1:40747 timed out after 1.814s (SENT)
I20260808 20:14:55.612042 13016 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:55.582573 15892 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:55.584594 16400 raft_consensus.cc:686] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 10 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:55.566506 11894 catalog_manager.cc:5495] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 2 to 5, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 5 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } }
I20260808 20:14:55.629914 11894 catalog_manager.cc:5495] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 1 to 3, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 3 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } }
I20260808 20:14:55.613394 16399 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:55.577764 16178 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:55.624243 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.601455 15674 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:55.617573 12254 raft_consensus.cc:2396] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 20:14:55.621943 16286 leader_election.cc:288] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:55.640399 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.647267 11894 catalog_manager.cc:5495] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:14:55.626368 15892 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:55.634738 16320 raft_consensus.cc:2781] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:14:55.497277 16312 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:55.648664 16178 raft_consensus.cc:3037] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:55.636122 16306 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:55.690961 16320 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:55.733691 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:55.734577 12047 raft_consensus.cc:2362] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6. Re-sending same reply.
I20260808 20:14:55.711150 16320 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Advancing to term 8
W20260808 20:14:55.748708 12102 leader_election.cc:334] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Timed out: RequestConsensusVote RPC to 127.10.24.1:40747 timed out after 1.965s (SENT)
I20260808 20:14:55.749217 12102 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:55.761114 12152 raft_consensus.cc:2454] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6.
I20260808 20:14:55.652390 11894 catalog_manager.cc:5495] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 2 to 4, leader changed from e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 4 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260808 20:14:55.771790 12102 leader_election.cc:334] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Timed out: RequestConsensusVote RPC to 127.10.24.1:40747 timed out after 2.013s (SENT)
I20260808 20:14:55.772886 11894 catalog_manager.cc:5495] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 2 to 6, leader changed from e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 6 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:55.776763 11894 catalog_manager.cc:5495] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:55.778190 11894 catalog_manager.cc:5495] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 1 to 3, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4). New cstate: current_term: 3 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:55.779804 11894 catalog_manager.cc:5495] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 1 to 3, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 3 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } }
I20260808 20:14:55.781829 11894 catalog_manager.cc:5495] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 4 to 7, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 7 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:14:55.781169 13016 leader_election.cc:302] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:55.789160 16359 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:55.794167 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.854251 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:55.797612 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.855032 12043 raft_consensus.cc:2362] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4. Re-sending same reply.
I20260808 20:14:55.799464 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.855718 12039 raft_consensus.cc:2454] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 2.
I20260808 20:14:55.801366 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:55.802096 12147 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.976s user 0.003s sys 0.022s
W20260808 20:14:55.856546 12147 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.031s user 0.004s sys 0.022s
I20260808 20:14:55.803297 12250 raft_consensus.cc:2396] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260808 20:14:55.855216 12047 raft_consensus.cc:2454] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
W20260808 20:14:55.816264 12102 leader_election.cc:334] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Timed out: RequestConsensusVote RPC to 127.10.24.1:40747 timed out after 2.053s (SENT)
I20260808 20:14:55.858239 11894 catalog_manager.cc:5495] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 1 to 3, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4). New cstate: current_term: 3 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
W20260808 20:14:55.859011 12102 leader_election.cc:334] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Timed out: RequestConsensusVote RPC to 127.10.24.1:40747 timed out after 2.063s (SENT)
I20260808 20:14:55.859719 11894 catalog_manager.cc:5495] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:55.735989 16400 consensus_queue.cc:237] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.11, Last appended by leader: 11, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:55.885596 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:55.890391 12152 raft_consensus.cc:2396] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 20:14:55.892267 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.894310 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.879333 16240 raft_consensus.cc:2781] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election won for term 6
W20260808 20:14:55.893111 15931 consensus_meta.cc:287] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.612s user 0.005s sys 0.005s
I20260808 20:14:55.907802 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.908613 12038 raft_consensus.cc:2379] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 7: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:55.909325 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:55.910091 12041 raft_consensus.cc:2379] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 8: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:55.917311 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.918603 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:55.919994 12101 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.945s user 0.010s sys 0.014s
W20260808 20:14:55.920333 12101 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.945s user 0.010s sys 0.014s
I20260808 20:14:55.917948 12041 raft_consensus.cc:2454] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:55.911690 16240 raft_consensus.cc:686] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:55.923899 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:55.920640 12101 connection.cc:232] RPC call timeout handler was delayed by -2.3283308220561594s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1814s for a response
W20260808 20:14:55.929056 12101 connection.cc:232] RPC call timeout handler was delayed by -2.1132308219093829s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2053s for a response
I20260808 20:14:55.913682 16312 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:55.930300 12041 raft_consensus.cc:2379] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 5: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
W20260808 20:14:55.924836 16420 log.cc:924] Time spent T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.143s user 0.000s sys 0.001s
W20260808 20:14:55.929329 12101 connection.cc:232] RPC call timeout handler was delayed by -2.0376308220438659s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2137s for a response
I20260808 20:14:55.900314 15892 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:55.939049 12206 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.038s user 0.008s sys 0.018s
W20260808 20:14:55.939345 12244 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.115s user 0.012s sys 0.060s
W20260808 20:14:55.939689 12244 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.115s user 0.012s sys 0.060s
W20260808 20:14:55.936667 12101 connection.cc:232] RPC call timeout handler was delayed by -1.2810317908879369s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1908s for a response
W20260808 20:14:55.937124 13072 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.742s user 0.015s sys 0.011s
I20260808 20:14:55.911913 15931 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:55.941092 13072 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.746s user 0.015s sys 0.011s
W20260808 20:14:55.935209 12000 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.088s user 0.016s sys 0.010s
I20260808 20:14:55.937937 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:55.942392 12000 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.095s user 0.016s sys 0.010s
W20260808 20:14:55.942765 12000 connection.cc:232] RPC call timeout handler was delayed by -1.1095882570371032s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1723s for a response
W20260808 20:14:55.943121 12000 connection.cc:232] RPC call timeout handler was delayed by -1.0810198050457984s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2197s for a response
W20260808 20:14:55.943490 12000 connection.cc:232] RPC call timeout handler was delayed by -1.0276882571633905s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1814s for a response
W20260808 20:14:55.944268 13079 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.884s user 0.018s sys 0.009s
W20260808 20:14:55.944625 13079 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.884s user 0.018s sys 0.009s
I20260808 20:14:55.945415 13079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57330 (request call id 507) took 3179 ms. Trace:
I20260808 20:14:55.945818 13079 rpcz_store.cc:276] 0808 20:14:52.765755 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:52.772117 (+ 6362us) service_pool.cc:227] Handling call
0808 20:14:52.778013 (+ 5896us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:52.843006 (+ 64993us) raft_consensus.cc:1520] Early marking committed up to index 7
0808 20:14:55.059990 (+2216984us) log.cc:842] Serialized 14 byte log entry
0808 20:14:55.945080 (+885090us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:55.945127 (+ 47us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:14:55.945168 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:55.945251 (+ 83us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:55.945361 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":2728994,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":38,"thread_start_us":3038326,"threads_started":2}
W20260808 20:14:55.939340 12206 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.038s user 0.008s sys 0.018s
I20260808 20:14:55.942055 12047 raft_consensus.cc:2454] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
W20260808 20:14:55.949038 12105 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.987s user 0.012s sys 0.015s
W20260808 20:14:55.949373 12105 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.988s user 0.012s sys 0.015s
W20260808 20:14:55.949728 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 988270us
W20260808 20:14:55.950989 13074 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.787s user 0.022s sys 0.050s
W20260808 20:14:55.951283 13074 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.788s user 0.022s sys 0.050s
W20260808 20:14:55.940685 12101 connection.cc:232] RPC call timeout handler was delayed by -1.279995835153386s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16620000000000001s for a response
W20260808 20:14:55.952961 12101 connection.cc:232] RPC call timeout handler was delayed by -1.2297317909542471s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1965s for a response
W20260808 20:14:55.954434 12101 connection.cc:232] RPC call timeout handler was delayed by -1.1424317909404635s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2062s for a response
W20260808 20:14:55.955710 12101 connection.cc:232] RPC call timeout handler was delayed by -1.0254317908547819s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21920000000000003s for a response
I20260808 20:14:55.957103 12101 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:55.958932 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:55.944056 12000 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:55.959514 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.967041 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:55.967304 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:55.968817 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:55.969525 12248 raft_consensus.cc:2379] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:55.973512 13077 raft_consensus.cc:2379] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 7: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:55.967869 12254 raft_consensus.cc:2454] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:14:55.974642 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:55.975157 12254 raft_consensus.cc:2362] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 9. Re-sending same reply.
I20260808 20:14:55.975911 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:55.976490 12254 raft_consensus.cc:2362] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8. Re-sending same reply.
I20260808 20:14:55.962728 12000 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:55.979265 12101 leader_election.cc:302] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
W20260808 20:14:55.989221 12048 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.013s user 0.017s sys 0.014s
W20260808 20:14:55.990118 12048 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.014s user 0.017s sys 0.015s
I20260808 20:14:56.007161 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:55.988057 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:55.978245 12000 leader_election.cc:302] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:56.008103 12038 raft_consensus.cc:2379] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 7: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
W20260808 20:14:56.009727 12205 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.898s user 0.012s sys 0.028s
W20260808 20:14:56.010221 11994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.743s user 0.025s sys 0.074s
W20260808 20:14:56.057605 12205 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.952s user 0.013s sys 0.028s
I20260808 20:14:56.010860 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:56.057958 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 953478us
I20260808 20:14:56.011230 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:56.010897 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:56.011957 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 47654us
I20260808 20:14:56.013978 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:56.014173 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:55.957824 16334 raft_consensus.cc:2781] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:56.019062 12105 leader_election.cc:302] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
W20260808 20:14:56.062119 12141 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.004s sys 0.018s
W20260808 20:14:56.020783 12249 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.008s sys 0.008s
I20260808 20:14:56.021564 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:56.062445 12141 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.547s user 0.004s sys 0.018s
I20260808 20:14:56.017189 12000 leader_election.cc:302] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
W20260808 20:14:56.063086 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 45982.2us
W20260808 20:14:56.017639 12207 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.032s user 0.028s sys 0.081s
W20260808 20:14:56.063686 12207 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.078s user 0.028s sys 0.082s
W20260808 20:14:56.063963 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 1.07859e+06us
I20260808 20:14:55.987952 16341 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:56.030964 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:56.066973 12249 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.584s user 0.008s sys 0.008s
I20260808 20:14:56.032491 12038 raft_consensus.cc:1261] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 20:14:56.009554 16271 raft_consensus.cc:2781] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:14:56.033927 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:56.067648 12039 raft_consensus.cc:2379] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 7: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
W20260808 20:14:56.034988 12138 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.000s sys 0.018s
W20260808 20:14:56.035360 12144 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.000s sys 0.017s
W20260808 20:14:56.039366 12001 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.619s user 0.008s sys 0.009s
I20260808 20:14:55.942459 16388 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:56.015890 16372 raft_consensus.cc:2781] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:56.040915 12048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 581) took 5153 ms. Trace:
I20260808 20:14:56.073734 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:56.042640 12142 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.005s sys 0.012s
W20260808 20:14:56.078931 12142 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.578s user 0.005s sys 0.012s
I20260808 20:14:56.018610 15826 raft_consensus.cc:2781] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:56.043490 12045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 583) took 2449 ms. Trace:
W20260808 20:14:56.044152 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 487100us
I20260808 20:14:56.045463 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:56.046123 12140 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.007s sys 0.011s
W20260808 20:14:56.056973 12150 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.020s sys 0.033s
I20260808 20:14:56.058609 12205 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
W20260808 20:14:56.057732 11994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.790s user 0.025s sys 0.074s
W20260808 20:14:56.089910 11994 connection.cc:232] RPC call timeout handler was delayed by -3.2854164859745651s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16810000000000003s for a response
W20260808 20:14:56.090296 11994 connection.cc:232] RPC call timeout handler was delayed by -3.2593379800673574s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.209s for a response
W20260808 20:14:56.090605 11994 connection.cc:232] RPC call timeout handler was delayed by -3.2403037990443408s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2069s for a response
I20260808 20:14:56.059063 12101 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:56.091526 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:56.091943 12101 leader_election.cc:302] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:56.059922 12047 raft_consensus.cc:2454] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:14:56.062000 12105 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:55.972461 16399 raft_consensus.cc:2781] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:56.069410 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:56.069830 12001 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.650s user 0.008s sys 0.009s
W20260808 20:14:56.070039 12144 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.600s user 0.000s sys 0.017s
W20260808 20:14:56.100526 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 680604us
W20260808 20:14:56.069412 12138 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.598s user 0.001s sys 0.018s
I20260808 20:14:56.072835 12041 raft_consensus.cc:2396] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260808 20:14:56.058991 12243 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
W20260808 20:14:56.055040 16424 log.cc:924] Time spent T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.150s user 0.001s sys 0.000s
W20260808 20:14:56.041925 12245 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.003s sys 0.014s
W20260808 20:14:56.108337 12245 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.611s user 0.003s sys 0.014s
W20260808 20:14:56.060968 15592 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.507s user 0.001s sys 0.029s
I20260808 20:14:56.082161 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:56.103223 11872 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet cb8b79bb1c59483e93af813653a73a31 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica b9d94f35ea15421aa1d721201c782d8a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:56.082862 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:56.087927 12140 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.600s user 0.007s sys 0.011s
W20260808 20:14:56.088025 12150 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.576s user 0.021s sys 0.033s
I20260808 20:14:56.019767 16240 consensus_queue.cc:237] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:56.090859 11994 connection.cc:232] RPC call timeout handler was delayed by -3.1630379802081734s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2197s for a response
I20260808 20:14:56.092095 13079 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 7.
I20260808 20:14:56.072518 16271 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:56.098269 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:56.115518 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:56.115717 12248 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8.
I20260808 20:14:56.095714 12000 leader_election.cc:302] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:56.072749 15674 raft_consensus.cc:2781] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 20:14:56.083666 12045 rpcz_store.cc:276] 0808 20:14:53.594517 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.594650 (+ 133us) service_pool.cc:227] Handling call
0808 20:14:53.622098 (+ 27448us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:53.622472 (+ 374us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:53.622501 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:53.623041 (+ 540us) log.cc:842] Serialized 26 byte log entry
0808 20:14:55.373032 (+1749991us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:55.373074 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:55.373141 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:56.043279 (+670138us) raft_consensus.cc:1671] finished
0808 20:14:56.043300 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:56.043434 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":86683,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":130,"thread_start_us":1749569,"threads_started":1,"wal-append.queue_time_us":2112987}
I20260808 20:14:56.109787 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:56.122512 11994 connection.cc:232] RPC call timeout handler was delayed by -3.0460379801224917s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23270000000000002s for a response
I20260808 20:14:56.113652 12001 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:56.083266 16372 raft_consensus.cc:482] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:56.128500 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:56.129093 12243 raft_consensus.cc:2454] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:56.111174 12254 raft_consensus.cc:2454] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:14:56.083106 16359 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:56.073534 16357 raft_consensus.cc:2781] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:56.133335 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:56.079036 12048 rpcz_store.cc:276] 0808 20:14:50.887816 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:50.909445 (+ 21629us) service_pool.cc:227] Handling call
0808 20:14:52.349697 (+1440252us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:52.349912 (+ 215us) raft_consensus.cc:1520] Early marking committed up to index 8
0808 20:14:54.975796 (+2625884us) log.cc:842] Serialized 14 byte log entry
0808 20:14:55.991175 (+1015379us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:55.991846 (+ 671us) raft_consensus.cc:1631] Marking committed up to 8
0808 20:14:55.991894 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:55.991958 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:56.040720 (+ 48762us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b5400170550. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d9a09 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:56.040839 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":3087246,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":2590,"raft.queue_time_us":98,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":5466,"spinlock_wait_cycles":3311572736,"thread_start_us":3640451,"threads_started":2}
W20260808 20:14:56.124697 11994 connection.cc:232] RPC call timeout handler was delayed by -2.815088507020846s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21440000000000003s for a response
W20260808 20:14:56.134807 11994 connection.cc:232] RPC call timeout handler was delayed by -2.8035831400193274s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15400000000000003s for a response
I20260808 20:14:56.109335 16201 raft_consensus.cc:2781] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260808 20:14:56.125346 15666 consensus_meta.cc:287] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.563s user 0.005s sys 0.005s
I20260808 20:14:56.121685 16271 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:56.129595 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:56.130438 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:56.060391 16178 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:56.112845 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 86473.6us
W20260808 20:14:56.135572 11994 connection.cc:232] RPC call timeout handler was delayed by -2.6909221729729325s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1796s for a response
W20260808 20:14:56.186489 11994 connection.cc:232] RPC call timeout handler was delayed by -2.6118831399362534s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1753s for a response
W20260808 20:14:56.186896 11994 connection.cc:232] RPC call timeout handler was delayed by -2.5793221730273217s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.192s for a response
W20260808 20:14:56.187207 11994 connection.cc:232] RPC call timeout handler was delayed by -2.5578831399325281s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18130000000000002s for a response
I20260808 20:14:56.136277 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:56.137364 12248 raft_consensus.cc:2379] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:56.138916 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:56.132978 12000 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:56.111222 12244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1851) took 5815 ms. Trace:
I20260808 20:14:56.107496 16399 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:56.139945 13074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53296 (request call id 758) took 1075 ms. Trace:
I20260808 20:14:56.078202 16396 consensus_queue.cc:1035] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.050s
I20260808 20:14:56.127369 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:56.134137 16372 raft_consensus.cc:3037] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:56.077225 15908 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:56.187464 11994 connection.cc:232] RPC call timeout handler was delayed by -2.5055221728980541s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2002s for a response
W20260808 20:14:56.204407 11994 connection.cc:232] RPC call timeout handler was delayed by -2.4597831398714334s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1922s for a response
W20260808 20:14:56.204743 11994 connection.cc:232] RPC call timeout handler was delayed by -2.2095831399783492s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22000000000000003s for a response
I20260808 20:14:56.196444 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:56.188221 12254 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
I20260808 20:14:56.081761 16341 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:56.130039 15892 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:56.191880 13074 rpcz_store.cc:276] 0808 20:14:55.064786 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:55.118086 (+ 53300us) service_pool.cc:227] Handling call
0808 20:14:55.163252 (+ 45166us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:55.163467 (+ 215us) raft_consensus.cc:1520] Early marking committed up to index 7
0808 20:14:55.951582 (+788115us) log.cc:842] Serialized 14 byte log entry
0808 20:14:56.139620 (+188038us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:56.139669 (+ 49us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:14:56.139713 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:56.139783 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:56.139890 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1657,"prepare.queue_time_us":408757,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"thread_start_us":973864,"threads_started":2}
W20260808 20:14:56.192836 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 59957.8us
I20260808 20:14:56.126158 16428 raft_consensus.cc:2781] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:56.189359 12244 rpcz_store.cc:276] 0808 20:14:50.295728 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:54.824028 (+4528300us) service_pool.cc:227] Handling call
0808 20:14:54.824297 (+ 269us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:54.824507 (+ 210us) raft_consensus.cc:1520] Early marking committed up to index 4
0808 20:14:55.939997 (+1115490us) log.cc:842] Serialized 14 byte log entry
0808 20:14:56.110896 (+170899us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:56.110950 (+ 54us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:14:56.111001 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:56.111064 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:56.111168 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1258936,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":2400,"raft.queue_time_us":10556,"spinlock_wait_cycles":21823744,"thread_start_us":1284093,"threads_started":2}
I20260808 20:14:56.195163 12241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 507) took 6091 ms. Trace:
I20260808 20:14:56.133816 16357 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:56.225735 16357 raft_consensus.cc:3037] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:56.226972 12251 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 518) took 1339 ms. Trace:
I20260808 20:14:56.228760 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:56.197876 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:56.133533 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:56.205108 11994 connection.cc:232] RPC call timeout handler was delayed by -2.1062266509979963s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2911s for a response
I20260808 20:14:56.116101 16286 raft_consensus.cc:2726] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:56.205873 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:56.211741 12000 leader_election.cc:302] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:56.214335 12241 rpcz_store.cc:276] 0808 20:14:50.104226 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.838923 (+3734697us) service_pool.cc:227] Handling call
0808 20:14:53.839264 (+ 341us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:53.839598 (+ 334us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:53.839634 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:53.868399 (+ 28765us) spinlock_profiling.cc:243] Waited 1.42 ms on lock 0x7b2c0011ebd8. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa966cbde11 00007fa966ccd442 00007fa968315c11 00007fa9682da58a 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772
0808 20:14:53.941263 (+ 72864us) log.cc:842] Serialized 26 byte log entry
0808 20:14:55.486100 (+1544837us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:55.486138 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:55.486199 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:56.144643 (+658444us) raft_consensus.cc:1671] finished
0808 20:14:56.144661 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:56.195105 (+ 50444us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2259,"raft.queue_time_us":1831,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":14729,"spinlock_wait_cycles":3270528,"thread_start_us":1441521,"threads_started":1,"wal-append.queue_time_us":1912677}
I20260808 20:14:56.213617 16428 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:56.216092 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:56.218814 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:56.219519 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:56.221298 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:56.250911 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:56.221521 16433 log.cc:924] Time spent T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.172s user 0.000s sys 0.000s
I20260808 20:14:56.222121 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:56.223148 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:56.223976 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:56.224643 16371 consensus_meta.cc:287] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.671s user 0.009s sys 0.000s
I20260808 20:14:56.225211 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:56.249223 16439 raft_consensus.cc:2781] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 20:14:56.195621 16399 raft_consensus.cc:3037] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:56.232304 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:56.235379 16438 raft_consensus.cc:2781] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:14:56.145539 16005 raft_consensus.cc:2781] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:14:56.145025 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
W20260808 20:14:56.236418 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 144541us
I20260808 20:14:56.275921 12101 leader_election.cc:302] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
W20260808 20:14:56.215746 16425 log.cc:924] Time spent T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.146s user 0.000s sys 0.000s
W20260808 20:14:56.240389 11994 connection.cc:232] RPC call timeout handler was delayed by -2.10463798022829s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33730000000000004s for a response
I20260808 20:14:56.227348 12251 rpcz_store.cc:276] 0808 20:14:54.887314 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:55.374094 (+486780us) service_pool.cc:227] Handling call
0808 20:14:55.374432 (+ 338us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:55.374774 (+ 342us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:55.374801 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:55.375347 (+ 546us) log.cc:842] Serialized 26 byte log entry
0808 20:14:55.912017 (+536670us) raft_consensus.cc:1631] Marking committed up to 8
0808 20:14:56.151136 (+239119us) log.cc:842] Serialized 14 byte log entry
0808 20:14:56.203836 (+ 52700us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:56.203916 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:56.226762 (+ 22846us) raft_consensus.cc:1671] finished
0808 20:14:56.226781 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:56.226898 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41258,"raft.queue_time_us":134557,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":20277,"thread_start_us":688312,"threads_started":2,"wal-append.queue_time_us":778745}
I20260808 20:14:56.135741 15931 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:56.245110 13071 raft_consensus.cc:2454] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:56.248958 12000 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:56.258700 16428 raft_consensus.cc:3037] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:56.249958 12244 raft_consensus.cc:2454] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 2.
I20260808 20:14:56.250730 12041 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:56.259831 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:56.303776 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:56.235764 16427 raft_consensus.cc:2781] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:56.261328 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:56.229341 12207 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:56.261965 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:56.196987 16008 consensus_queue.cc:1046] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260808 20:14:56.144315 15666 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:56.272436 12247 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:56.138827 16201 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:56.221522 16341 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:56.239670 16178 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:14:56.276225 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 162660us
I20260808 20:14:56.275944 12243 raft_consensus.cc:2396] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20260808 20:14:56.282325 11994 connection.cc:232] RPC call timeout handler was delayed by -1.9639808780048043s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1504s for a response
W20260808 20:14:56.337543 11994 connection.cc:232] RPC call timeout handler was delayed by -1.7857808780390769s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17020000000000002s for a response
W20260808 20:14:56.337904 11994 connection.cc:232] RPC call timeout handler was delayed by -1.7812808779999614s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17070000000000002s for a response
W20260808 20:14:56.338199 11994 connection.cc:232] RPC call timeout handler was delayed by -1.4590808779466897s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20650000000000002s for a response
W20260808 20:14:56.339438 11994 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.24.3:38391 timed out after 1.504s (SENT) [suppressed 47 similar messages]
W20260808 20:14:56.340157 11994 leader_election.cc:334] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Timed out: RequestConsensusVote RPC to 127.10.24.3:38391 timed out after 1.540s (SENT)
W20260808 20:14:56.282743 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 179918us
I20260808 20:14:56.301714 13014 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:56.253672 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 191551us
I20260808 20:14:56.344236 12105 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
W20260808 20:14:56.308638 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 79364.5us
I20260808 20:14:56.310969 12254 raft_consensus.cc:1226] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260808 20:14:56.273746 16438 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:56.321982 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:56.262004 16371 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:56.284473 16432 raft_consensus.cc:2781] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:56.340572 11994 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:56.348929 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 47542.1us
I20260808 20:14:56.309471 16427 raft_consensus.cc:482] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:14:56.350800 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 50680.6us
I20260808 20:14:56.332870 16201 raft_consensus.cc:3037] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:56.317673 16388 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 10 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:56.361130 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:56.375259 12247 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 8.
W20260808 20:14:56.376646 16454 log.cc:924] Time spent T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260808 20:14:56.377465 13014 leader_election.cc:302] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:56.378263 16501 raft_consensus.cc:2781] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:56.394150 16501 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:56.394560 16501 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
W20260808 20:14:56.394994 16423 log.cc:924] Time spent T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.602s user 0.005s sys 0.000s
I20260808 20:14:56.383599 16488 raft_consensus.cc:2781] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:56.396428 16488 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:56.360744 16480 raft_consensus.cc:2781] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:56.397269 16480 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:56.397574 16480 raft_consensus.cc:3037] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:56.347808 16422 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:56.396796 16488 raft_consensus.cc:3037] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:56.401180 16422 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:56.352764 16438 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:56.403167 15908 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:56.406118 16371 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:56.406864 16501 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:56.407507 15666 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:56.361618 16178 raft_consensus.cc:2726] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:56.381201 12138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 418) took 1269 ms. Trace:
I20260808 20:14:56.382763 15674 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:56.367697 12207 leader_election.cc:302] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:56.334199 16434 raft_consensus.cc:2726] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:56.417157 15674 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:14:56.384828 12147 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1758) took 5513 ms. Trace:
I20260808 20:14:56.417649 12147 rpcz_store.cc:276] 0808 20:14:50.871882 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.063505 (+2191623us) service_pool.cc:227] Handling call
0808 20:14:53.589440 (+525935us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:53.834522 (+245082us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:53.834548 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:53.867302 (+ 32754us) log.cc:842] Serialized 26 byte log entry
0808 20:14:55.856984 (+1989682us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:14:55.857029 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:55.857101 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:56.384627 (+527526us) raft_consensus.cc:1671] finished
0808 20:14:56.384648 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:56.384770 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":245417,"raft.queue_time_us":258979,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":12803,"thread_start_us":1031126,"threads_started":1,"wal-append.queue_time_us":1215558}
I20260808 20:14:56.393373 12250 raft_consensus.cc:2379] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 8: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:56.409704 12141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1774) took 1566 ms. Trace:
I20260808 20:14:56.419509 12141 rpcz_store.cc:276] 0808 20:14:54.843557 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:55.385094 (+541537us) service_pool.cc:227] Handling call
0808 20:14:55.514606 (+129512us) spinlock_profiling.cc:243] Waited 6.38 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:14:55.514814 (+ 208us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:55.515066 (+ 252us) raft_consensus.cc:1520] Early marking committed up to index 2
0808 20:14:55.515090 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:55.515523 (+ 433us) log.cc:842] Serialized 26 byte log entry
0808 20:14:56.062805 (+547282us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:14:56.062840 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:56.062904 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:56.409416 (+346512us) raft_consensus.cc:1671] finished
0808 20:14:56.409437 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:56.409642 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14678400,"thread_start_us":546942,"threads_started":1,"wal-append.queue_time_us":742469}
I20260808 20:14:56.416661 12138 rpcz_store.cc:276] 0808 20:14:55.111418 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:55.469842 (+358424us) service_pool.cc:227] Handling call
0808 20:14:55.470158 (+ 316us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:55.470422 (+ 264us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:55.470444 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:55.470825 (+ 381us) log.cc:842] Serialized 26 byte log entry
0808 20:14:56.101135 (+630310us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:56.101170 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:56.101226 (+ 56us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:56.381009 (+279783us) raft_consensus.cc:1671] finished
0808 20:14:56.381027 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:56.381145 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":83907,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":2745,"thread_start_us":629953,"threads_started":1,"wal-append.queue_time_us":908025}
I20260808 20:14:56.354269 16432 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:56.424194 16480 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:56.425173 15674 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:56.413230 12140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1756) took 5542 ms. Trace:
I20260808 20:14:56.424619 16432 raft_consensus.cc:3037] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
W20260808 20:14:56.425532 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 57921.9us
I20260808 20:14:56.408790 16501 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:56.353282 16427 raft_consensus.cc:3037] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:56.426074 12140 rpcz_store.cc:276] 0808 20:14:50.871339 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.043089 (+2171750us) service_pool.cc:227] Handling call
0808 20:14:53.591804 (+548715us) raft_consensus.cc:1457] Updating replica for 3 ops
0808 20:14:53.842521 (+250717us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:53.842550 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0808 20:14:53.904584 (+ 62034us) log.cc:842] Serialized 78 byte log entry
0808 20:14:55.430584 (+1526000us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:56.111608 (+681024us) log.cc:842] Serialized 14 byte log entry
0808 20:14:56.254093 (+142485us) log.cc:842] Serialized 14 byte log entry
0808 20:14:56.343759 (+ 89666us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:56.343840 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:56.394465 (+ 50625us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:14:56.413024 (+ 18559us) raft_consensus.cc:1671] finished
0808 20:14:56.413049 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:56.413180 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":251302,"prepare.queue_time_us":676727,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":65,"raft.queue_time_us":674229,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":66372,"spinlock_wait_cycles":23155456,"thread_start_us":2251030,"threads_started":4,"wal-append.queue_time_us":1851219}
I20260808 20:14:56.434312 13076 raft_consensus.cc:1226] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [term 10 LEADER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 9. Current term is 10. Ops: []
I20260808 20:14:56.435864 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:56.436501 12247 raft_consensus.cc:3037] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:56.437700 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:56.438339 12244 raft_consensus.cc:3037] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
W20260808 20:14:56.432523 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 88351us
I20260808 20:14:56.444052 12105 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:56.444840 12105 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:56.425688 16388 raft_consensus.cc:2726] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:56.427282 15674 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:56.472307 12144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53162 (request call id 1468) took 2627 ms. Trace:
I20260808 20:14:56.471688 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:56.481647 12105 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:56.483520 12254 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Advancing to term 10
W20260808 20:14:56.474129 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 125856us
I20260808 20:14:56.488389 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 419) took 1309 ms. Trace:
I20260808 20:14:56.472723 12144 rpcz_store.cc:276] 0808 20:14:53.845001 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:53.845129 (+ 128us) service_pool.cc:227] Handling call
0808 20:14:53.849004 (+ 3875us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 20:14:54.494080 (+645076us) spinlock_profiling.cc:243] Waited 37.4 ms on lock 0x7b480024c258. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96d1817f1 00007fa96d1788e7 00007fa96d17877a 00007fa96d26c95f 00007fa96d26c627 00007fa96d280b2c 00007fa96d280ad8 00007fa96d280a58 00007fa96d280a13 00007fa96d27fcbc 00007fa96c201be6
0808 20:14:55.418307 (+924227us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:55.418338 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 20:14:55.418870 (+ 532us) log.cc:842] Serialized 52 byte log entry
0808 20:14:56.100640 (+681770us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:14:56.249257 (+148617us) log.cc:842] Serialized 14 byte log entry
0808 20:14:56.447218 (+197961us) log.cc:842] Serialized 14 byte log entry
0808 20:14:56.447435 (+ 217us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:56.447500 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:56.472087 (+ 24587us) raft_consensus.cc:1671] finished
0808 20:14:56.472109 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:56.472249 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6629,"prepare.queue_time_us":117563,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":71,"raft.queue_time_us":151316,"raft.run_cpu_time_us":2679,"raft.run_wall_time_us":620012,"spinlock_wait_cycles":85905664,"thread_start_us":2514076,"threads_started":4,"wal-append.queue_time_us":862851}
I20260808 20:14:56.461402 16501 raft_consensus.cc:2781] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:56.426188 16434 raft_consensus.cc:2781] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260808 20:14:56.462666 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 213827us
W20260808 20:14:56.496757 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 220901us
I20260808 20:14:56.466884 16439 raft_consensus.cc:686] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:56.519147 12101 leader_election.cc:302] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:56.517110 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:56.517335 12001 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:56.517843 12244 raft_consensus.cc:2454] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
I20260808 20:14:56.518500 12000 leader_election.cc:302] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:56.482597 16008 raft_consensus.cc:2781] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:56.523129 12101 leader_election.cc:302] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
W20260808 20:14:56.523566 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 163617us
W20260808 20:14:56.524740 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 184598us
I20260808 20:14:56.443193 16372 raft_consensus.cc:504] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:56.529831 12241 raft_consensus.cc:2379] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:56.444275 16271 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:56.528057 12101 leader_election.cc:302] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
W20260808 20:14:56.527743 11994 leader_election.cc:334] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Timed out: RequestConsensusVote RPC to 127.10.24.3:38391 timed out after 1.681s (SENT)
I20260808 20:14:56.532650 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:56.533281 12254 raft_consensus.cc:2454] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 10.
W20260808 20:14:56.534613 11994 leader_election.cc:334] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Timed out: RequestConsensusVote RPC to 127.10.24.3:38391 timed out after 1.753s (SENT)
I20260808 20:14:56.535107 11994 leader_election.cc:302] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:56.536190 11994 leader_election.cc:334] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Timed out: RequestConsensusVote RPC to 127.10.24.3:38391 timed out after 1.796s (SENT)
I20260808 20:14:56.536624 11994 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:56.488806 12142 rpcz_store.cc:276] 0808 20:14:55.178535 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:55.500187 (+321652us) service_pool.cc:227] Handling call
0808 20:14:55.500468 (+ 281us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:55.500654 (+ 186us) raft_consensus.cc:1520] Early marking committed up to index 7
0808 20:14:56.079425 (+578771us) log.cc:842] Serialized 14 byte log entry
0808 20:14:56.488063 (+408638us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:56.488117 (+ 54us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:14:56.488158 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:56.488224 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:56.488331 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":641960,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":3599,"raft.queue_time_us":193543,"raft.run_cpu_time_us":66,"raft.run_wall_time_us":63,"thread_start_us":986012,"threads_started":3,"wal-append.queue_time_us":187770}
I20260808 20:14:56.525221 12247 raft_consensus.cc:2454] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 7.
I20260808 20:14:56.535004 13014 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:56.473112 15699 consensus_queue.cc:1035] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.050s
I20260808 20:14:56.519737 16434 raft_consensus.cc:482] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:56.521916 15674 raft_consensus.cc:2781] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:56.514146 16501 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:56.496577 16477 raft_consensus.cc:2781] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:14:56.473363 16341 consensus_queue.cc:1046] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.057s
I20260808 20:14:56.559274 12045 raft_consensus.cc:2379] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 8: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:56.559741 13014 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:56.560441 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:56.562877 12101 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:56.526718 16008 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:56.575201 12101 leader_election.cc:302] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:56.576427 12101 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:56.583036 12144 raft_consensus.cc:1226] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 10 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 9. Current term is 10. Ops: [9.11-9.11]
W20260808 20:14:56.584304 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 48139.9us
I20260808 20:14:56.541013 16503 raft_consensus.cc:2781] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:56.542397 16400 raft_consensus.cc:2781] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 20:14:56.525996 16497 raft_consensus.cc:2781] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:56.587069 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:56.615948 12150 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 8.
I20260808 20:14:56.584779 15666 raft_consensus.cc:2781] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 20:14:56.561798 16501 raft_consensus.cc:3037] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:56.559247 16475 raft_consensus.cc:2726] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:56.593747 16341 consensus_queue.cc:1225] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer e5bd9a4616ce4acfa20dc7a002984a20 log is divergent from this leader: its last log entry 10.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260808 20:14:56.557170 15674 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:56.558842 16434 raft_consensus.cc:3037] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:56.561486 15699 raft_consensus.cc:2781] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 20:14:56.561875 16477 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:56.581054 16008 raft_consensus.cc:3037] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:56.606863 16480 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:56.593660 16503 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:56.616387 16497 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:14:56.617754 16320 consensus_meta.cc:287] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.857s user 0.011s sys 0.000s
I20260808 20:14:56.637811 16520 raft_consensus.cc:3032] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 20:14:56.638259 16520 raft_consensus.cc:729] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:56.621011 16400 raft_consensus.cc:686] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 10 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
W20260808 20:14:56.642771 16478 log.cc:924] Time spent T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.098s user 0.001s sys 0.000s
I20260808 20:14:56.642527 16399 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:56.639427 16503 raft_consensus.cc:3037] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:56.646593 16372 leader_election.cc:288] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:56.647990 16438 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:56.639061 16520 consensus_queue.cc:260] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:56.652881 16357 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:56.635386 16477 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:56.653755 16488 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:56.654518 16480 raft_consensus.cc:2781] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:56.654958 16511 raft_consensus.cc:2781] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:56.655853 16504 raft_consensus.cc:2781] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:56.656266 16513 raft_consensus.cc:2726] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:56.672353 16480 raft_consensus.cc:482] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:56.672735 16504 raft_consensus.cc:482] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:56.673220 16504 raft_consensus.cc:3037] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:56.673380 16520 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Advancing to term 10
I20260808 20:14:56.672194 16511 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:56.676201 16511 raft_consensus.cc:3037] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:56.678301 16488 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:56.640760 16320 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:56.672828 16480 raft_consensus.cc:3037] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:56.639674 16497 raft_consensus.cc:3037] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:56.686599 16509 raft_consensus.cc:2781] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:56.687050 16509 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:56.687395 16509 raft_consensus.cc:3037] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:56.689226 16508 raft_consensus.cc:3037] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:56.691573 16432 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:56.663060 16399 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:56.615239 16439 consensus_queue.cc:237] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:56.695343 16438 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:56.693317 16399 raft_consensus.cc:2726] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260808 20:14:56.585914 11994 leader_election.cc:334] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Timed out: RequestConsensusVote RPC to 127.10.24.3:38391 timed out after 1.920s (SENT)
I20260808 20:14:56.659471 16008 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:56.781899 11994 leader_election.cc:302] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:56.783990 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 198091us
I20260808 20:14:56.784279 16008 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:56.785269 16399 raft_consensus.cc:2726] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:56.740029 12048 raft_consensus.cc:2379] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:56.739734 12043 raft_consensus.cc:2454] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
W20260808 20:14:56.745388 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 224061us
W20260808 20:14:56.750461 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 220753us
I20260808 20:14:56.635778 15699 raft_consensus.cc:482] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:56.791808 15699 raft_consensus.cc:3037] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 20:14:56.606736 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:56.793656 12152 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 7.
I20260808 20:14:56.795295 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:56.612216 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:56.798316 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:56.799719 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:56.800521 12140 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:56.614432 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:14:56.801463 11994 leader_election.cc:334] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Timed out: RequestConsensusVote RPC to 127.10.24.3:38391 timed out after 2.002s (SENT)
I20260808 20:14:56.802417 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:56.803683 12141 raft_consensus.cc:2379] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 10: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:56.614401 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:56.792464 12102 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:56.754555 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 178047us
I20260808 20:14:56.611918 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:56.806174 12138 raft_consensus.cc:2454] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:56.807384 12101 leader_election.cc:302] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:56.808086 12138 raft_consensus.cc:1226] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260808 20:14:56.809317 12250 raft_consensus.cc:3032] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 9 LEADER]: Stepping down as leader of term 9
I20260808 20:14:56.755972 12147 raft_consensus.cc:1226] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7. Ops: [6.5-6.5]
W20260808 20:14:56.758533 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 241083us
I20260808 20:14:56.759128 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:56.759925 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:56.760763 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:56.693135 16499 raft_consensus.cc:2781] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:56.766522 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:14:56.623553 12206 connection.cc:232] RPC call timeout handler was delayed by -4.1425348350312561s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3018s for a response
I20260808 20:14:56.771270 12245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2009) took 5944 ms. Trace:
I20260808 20:14:56.649060 13072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53296 (request call id 753) took 1684 ms. Trace:
I20260808 20:14:56.640096 15666 raft_consensus.cc:686] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:56.628039 15674 raft_consensus.cc:3037] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:56.779167 16427 raft_consensus.cc:504] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:56.780088 16357 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:56.708163 16503 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:56.709697 16504 raft_consensus.cc:504] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:56.696085 16507 raft_consensus.cc:2726] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 20:14:56.709434 16497 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:56.763118 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:56.809830 12250 raft_consensus.cc:729] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:56.804589 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:56.800568 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:56.710320 16501 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:56.818033 16399 consensus_queue.cc:1035] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 20:14:56.818802 16271 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:56.665675 16201 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:56.841303 12146 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Advancing to term 9
W20260808 20:14:56.821053 13077 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57330 (request call id 517) took 1697 ms (client timeout 1818 ms). Trace:
I20260808 20:14:56.851323 16504 leader_election.cc:288] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:56.850943 12250 consensus_queue.cc:260] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 9.11, Last appended by leader: 11, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:56.852401 12206 connection.cc:232] RPC call timeout handler was delayed by -2.8022736839484423s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1521s for a response
I20260808 20:14:56.853632 16503 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:56.819316 16513 raft_consensus.cc:2726] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260808 20:14:56.861063 12206 connection.cc:232] RPC call timeout handler was delayed by -2.7329736838582903s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15980000000000003s for a response
W20260808 20:14:56.866055 12206 connection.cc:232] RPC call timeout handler was delayed by -2.3090736838057637s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2069s for a response
W20260808 20:14:56.858511 13077 rpcz_store.cc:269] 0808 20:14:55.122939 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:55.127336 (+ 4397us) service_pool.cc:227] Handling call
0808 20:14:56.820342 (+1693006us) spinlock_profiling.cc:243] Waited 798 ms on lock 0x7b5400030554. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:14:56.820690 (+ 348us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1836020608}
I20260808 20:14:56.869488 12150 raft_consensus.cc:2396] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 9 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260808 20:14:56.852864 13072 rpcz_store.cc:276] 0808 20:14:54.964685 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:54.977731 (+ 13046us) service_pool.cc:227] Handling call
0808 20:14:54.978106 (+ 375us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 20:14:55.156658 (+178552us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:55.156690 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 20:14:55.157247 (+ 557us) log.cc:842] Serialized 52 byte log entry
0808 20:14:55.941563 (+784316us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:14:55.941604 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:56.039056 (+ 97452us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b5400710f54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:56.039106 (+ 50us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:14:56.648752 (+609646us) raft_consensus.cc:1671] finished
0808 20:14:56.648774 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:56.648996 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":797,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":136,"spinlock_wait_cycles":409681664,"thread_start_us":746716,"threads_started":1,"wal-append.queue_time_us":1469432}
I20260808 20:14:56.870489 16499 raft_consensus.cc:482] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:56.871320 12250 raft_consensus.cc:3037] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Advancing to term 10
I20260808 20:14:56.872283 16201 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:56.837270 15931 raft_consensus.cc:482] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:56.877651 15931 raft_consensus.cc:504] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:56.662370 16400 consensus_queue.cc:237] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:56.884460 16334 raft_consensus.cc:482] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:56.885083 16334 raft_consensus.cc:3037] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:56.883308 16005 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:56.889222 16005 raft_consensus.cc:3037] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Advancing to term 8
W20260808 20:14:56.890753 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 67028.2us
I20260808 20:14:56.890552 16480 raft_consensus.cc:504] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:56.896476 16422 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:56.897400 12144 raft_consensus.cc:2379] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 5: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:56.665526 16434 raft_consensus.cc:504] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:56.696849 16428 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:56.697937 16432 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:56.846412 12245 rpcz_store.cc:276] 0808 20:14:50.826277 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:55.155371 (+4329094us) service_pool.cc:227] Handling call
0808 20:14:55.169196 (+ 13825us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:14:55.488620 (+319424us) raft_consensus.cc:1520] Early marking committed up to index 7
0808 20:14:56.108680 (+620060us) log.cc:842] Serialized 14 byte log entry
0808 20:14:56.305580 (+196900us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:14:56.305636 (+ 56us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:14:56.305678 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:14:56.770640 (+464962us) spinlock_profiling.cc:243] Waited 319 ms on lock 0x7b5400374e54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:14:56.770724 (+ 84us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:14:56.771052 (+ 328us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31035,"prepare.queue_time_us":715340,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":30542,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":14984,"spinlock_wait_cycles":734102784,"thread_start_us":779881,"threads_started":2,"wal-append.queue_time_us":352478}
W20260808 20:14:56.906752 12206 connection.cc:232] RPC call timeout handler was delayed by -2.0318736839108169s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2377s for a response
I20260808 20:14:56.834826 16521 raft_consensus.cc:2726] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:56.681144 16320 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:56.871194 16499 raft_consensus.cc:3037] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:56.918208 12101 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:56.883329 15666 consensus_queue.cc:237] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:56.932211 16501 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:56.933275 12206 connection.cc:232] RPC call timeout handler was delayed by -1.7528736838139594s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2687s for a response
W20260808 20:14:56.936250 12206 connection.cc:232] RPC call timeout handler was delayed by -1.1714736837893724s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33330000000000004s for a response
I20260808 20:14:56.939558 16427 leader_election.cc:288] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:56.940467 16497 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:56.946643 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 64473.2us
I20260808 20:14:56.934813 16480 leader_election.cc:288] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:56.946741 12102 leader_election.cc:302] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:14:56.928486 16477 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:56.924710 16511 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:57.001820 12146 raft_consensus.cc:2396] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
W20260808 20:14:56.984429 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 48198.5us
I20260808 20:14:57.010850 12101 leader_election.cc:302] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:57.023988 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:56.987823 12250 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 10 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 9 index: 11. Preceding OpId from leader: term: 10 index: 12. (index mismatch)
I20260808 20:14:56.983799 16427 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:14:57.024669 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:56.992336 16503 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:56.997550 15931 leader_election.cc:288] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:57.009191 16521 raft_consensus.cc:2781] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:56.990521 16508 raft_consensus.cc:2726] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 20:14:57.085441 15592 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20260808 20:14:57.085731 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:57.086126 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:57.086131 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:57.088490 16480 raft_consensus.cc:2781] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:57.089130 16428 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:57.086316 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 152855us
I20260808 20:14:57.095350 16240 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.013s
W20260808 20:14:57.095965 15826 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.933s user 0.029s sys 0.023s
I20260808 20:14:57.095803 16503 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:57.100409 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:57.093681 12206 leader_election.cc:302] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:57.092036 16534 log.cc:924] Time spent T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260808 20:14:57.080066 16422 raft_consensus.cc:2726] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 20:14:57.016623 16005 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:57.005559 15699 raft_consensus.cc:504] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:57.002462 15674 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:57.096662 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:57.102120 16511 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:57.078384 16427 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:56.997795 16509 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:57.095484 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:57.055817 16499 raft_consensus.cc:504] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:57.158525 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:57.161005 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 210589us
I20260808 20:14:57.153856 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:14:57.170326 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 159676us
I20260808 20:14:57.090209 16521 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:57.159034 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
W20260808 20:14:57.134045 15826 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.004s user 0.030s sys 0.023s
I20260808 20:14:57.102908 16480 raft_consensus.cc:482] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:57.160997 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:57.094404 16434 leader_election.cc:288] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:57.189747 12137 raft_consensus.cc:2379] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:57.189810 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:57.193199 12038 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 7.
I20260808 20:14:57.193516 16434 raft_consensus.cc:2726] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:57.194111 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:57.194726 15674 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:57.195603 13077 raft_consensus.cc:2454] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:57.194741 13080 raft_consensus.cc:3037] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:57.198259 16499 leader_election.cc:288] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:57.193856 12041 raft_consensus.cc:1261] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260808 20:14:57.199723 12047 raft_consensus.cc:2379] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 6: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:57.188236 12251 raft_consensus.cc:2454] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
W20260808 20:14:57.193760 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 139263us
I20260808 20:14:57.201129 12039 raft_consensus.cc:2379] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 7: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:57.109915 16439 consensus_queue.cc:1035] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.251s
I20260808 20:14:57.205256 16005 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:57.245443 13080 raft_consensus.cc:2454] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:14:57.207541 16503 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:57.209168 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:57.209787 13076 raft_consensus.cc:2354] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 7. Current term is 8.
I20260808 20:14:57.211318 12206 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:57.152848 16334 raft_consensus.cc:504] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:57.216382 16427 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:57.217017 15592 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:57.188861 15826 raft_consensus.cc:686] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:57.218269 16509 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:57.224763 16271 consensus_queue.cc:1035] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 20:14:57.225536 16499 consensus_queue.cc:1035] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:14:57.207922 12101 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:57.226318 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:57.227319 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:57.227586 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:57.227335 15699 leader_election.cc:288] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:57.230830 13075 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97"
dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:51996
I20260808 20:14:57.261924 13075 raft_consensus.cc:595] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:57.262280 13075 raft_consensus.cc:603] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 20:14:57.234592 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:57.236971 12105 leader_election.cc:302] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:57.239961 11994 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:57.240542 16508 consensus_queue.cc:1046] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 20:14:57.241439 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:57.190268 16480 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:57.267158 16306 raft_consensus.cc:2726] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:57.267220 13073 raft_consensus.cc:3037] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:57.268312 13072 raft_consensus.cc:1226] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:57.265374 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:57.268764 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:57.188324 16521 raft_consensus.cc:3037] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:57.257617 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
W20260808 20:14:57.262223 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 54456.8us
I20260808 20:14:57.264120 12140 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 8.
I20260808 20:14:57.272353 13078 raft_consensus.cc:3037] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:57.272732 16511 raft_consensus.cc:2726] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:57.273155 15592 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:57.272485 15674 raft_consensus.cc:2741] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260808 20:14:57.281102 13074 raft_consensus.cc:2379] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 6: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:57.272536 12046 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 7.
W20260808 20:14:57.290081 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 50190.9us
I20260808 20:14:57.269438 16334 leader_election.cc:288] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:57.292550 12251 raft_consensus.cc:3037] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:57.297199 15592 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:57.271205 16499 consensus_queue.cc:1225] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer b9d94f35ea15421aa1d721201c782d8a log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260808 20:14:57.288779 16427 raft_consensus.cc:3032] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 20:14:57.299150 13080 raft_consensus.cc:3037] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:57.300097 13014 leader_election.cc:302] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:57.300004 12047 raft_consensus.cc:2379] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 7: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:57.301157 16508 raft_consensus.cc:2781] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 20:14:57.302299 11994 leader_election.cc:302] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:57.303438 12250 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:14:57.267856 15826 consensus_queue.cc:237] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:57.299031 13071 raft_consensus.cc:2379] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 5: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:57.333801 13078 raft_consensus.cc:2454] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:14:57.320675 12251 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 8.
I20260808 20:14:57.314574 12250 raft_consensus.cc:595] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:57.339548 12250 raft_consensus.cc:603] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 20:14:57.321075 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 136960us
W20260808 20:14:57.324242 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 113081us
I20260808 20:14:57.338668 13073 raft_consensus.cc:2454] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:57.323814 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:57.341475 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:57.348258 12047 raft_consensus.cc:2379] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 8: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:57.349337 16439 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:57.350044 16439 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:57.352380 16439 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:57.354480 12241 raft_consensus.cc:1261] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 20:14:57.354921 12249 raft_consensus.cc:2454] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:57.304943 16427 raft_consensus.cc:729] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:57.345431 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:57.357542 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:57.325114 15892 raft_consensus.cc:2781] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:14:57.348124 12105 leader_election.cc:302] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:57.333356 16508 raft_consensus.cc:686] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:57.359457 15892 raft_consensus.cc:482] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:57.360008 15892 raft_consensus.cc:3037] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:57.316260 16499 raft_consensus.cc:2726] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:57.364405 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:57.360543 16508 consensus_queue.cc:237] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:57.379362 13014 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:57.380560 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:57.383775 12247 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:14:57.341878 16521 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:57.388154 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:57.384318 12247 raft_consensus.cc:595] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:14:57.388998 12247 raft_consensus.cc:603] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 20:14:57.394636 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 92457.6us
I20260808 20:14:57.396863 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:57.385149 15674 raft_consensus.cc:2781] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:57.399063 12102 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:57.400496 12105 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:57.390285 16240 raft_consensus.cc:2781] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 20:14:57.400784 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:57.409806 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:57.410315 13080 raft_consensus.cc:2454] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:14:57.398608 15674 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:14:57.421357 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 62114.3us
I20260808 20:14:57.425316 16516 raft_consensus.cc:2781] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:57.425504 12206 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:57.426487 16306 raft_consensus.cc:2726] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 20:14:57.428222 16306 consensus_queue.cc:1035] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:14:57.428782 16477 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:57.429301 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:57.416469 16240 raft_consensus.cc:686] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:57.429952 12243 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:14:57.430601 16240 consensus_queue.cc:237] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:57.410614 15699 consensus_queue.cc:1035] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260808 20:14:57.421439 12241 raft_consensus.cc:3037] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:57.422392 12251 raft_consensus.cc:1261] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 10 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260808 20:14:57.462482 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:57.428957 16306 consensus_queue.cc:1225] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer 78b9263889cd4f1582578a3aa4f644d9 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260808 20:14:57.433949 15674 raft_consensus.cc:3037] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:57.466156 12046 raft_consensus.cc:2379] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 7: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:57.470032 12206 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:57.470309 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:57.471045 12047 raft_consensus.cc:2354] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 4.
I20260808 20:14:57.461983 15826 raft_consensus.cc:3037] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:57.467136 12247 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e256c29dec964f49856cf204f950793b"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:14:57.468598 12207 leader_election.cc:302] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:57.477744 12101 leader_election.cc:302] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:57.481329 12105 leader_election.cc:398] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 election: Vote denied by peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901) with higher term. Message: Invalid argument: T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 7. Current term is 8.
I20260808 20:14:57.477492 16005 consensus_queue.cc:1035] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.008s
I20260808 20:14:57.481717 12105 leader_election.cc:401] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
W20260808 20:14:57.482208 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 44661.4us
I20260808 20:14:57.483105 12105 leader_election.cc:302] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:57.484236 12105 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:14:57.437158 16516 raft_consensus.cc:686] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:57.478683 12247 raft_consensus.cc:595] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:57.488448 12247 raft_consensus.cc:603] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 20:14:57.485227 16542 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Advancing to term 8
W20260808 20:14:57.487481 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 91936.2us
I20260808 20:14:57.489965 12241 raft_consensus.cc:2454] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:14:57.493150 12102 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:57.485211 16428 raft_consensus.cc:2781] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 20:14:57.460218 16511 consensus_queue.cc:1035] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.054s
I20260808 20:14:57.496789 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:57.489157 16422 raft_consensus.cc:2781] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:57.492998 15592 raft_consensus.cc:2741] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260808 20:14:57.501358 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260808 20:14:57.497573 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:57.502574 13071 raft_consensus.cc:2379] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 8: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
W20260808 20:14:57.498498 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 110066us
I20260808 20:14:57.500003 12046 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:57.433167 16427 consensus_queue.cc:260] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:57.507681 16422 raft_consensus.cc:686] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:57.508878 16501 raft_consensus.cc:2726] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:57.487974 16480 raft_consensus.cc:504] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:57.510452 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:57.510664 16432 raft_consensus.cc:2781] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:14:57.516439 16432 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:57.516829 16432 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:57.481287 15892 raft_consensus.cc:504] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:57.510957 16516 consensus_queue.cc:237] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:57.511718 15592 consensus_queue.cc:1046] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 20:14:57.510771 12102 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:57.513437 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:57.514822 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:57.515296 16306 raft_consensus.cc:2781] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:57.528343 12243 raft_consensus.cc:2454] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 9.
I20260808 20:14:57.528810 12251 raft_consensus.cc:2379] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 7: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:57.530118 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:57.531534 16422 consensus_queue.cc:237] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:57.532452 15592 raft_consensus.cc:3032] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:57.532912 15592 raft_consensus.cc:729] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:57.535778 12247 raft_consensus.cc:2379] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 9: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:57.523720 16521 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:57.540740 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:57.543937 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:57.546294 12102 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:57.546417 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:57.546802 12206 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:57.547916 12206 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:57.536451 16501 raft_consensus.cc:2726] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 20:14:57.549891 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:57.552089 12247 raft_consensus.cc:3037] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:57.553427 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:57.553840 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:57.554854 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:57.539659 15892 leader_election.cc:288] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:57.556715 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:57.557258 16511 raft_consensus.cc:2726] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 20:14:57.557255 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:57.559521 12251 raft_consensus.cc:3037] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:57.560343 12045 raft_consensus.cc:2379] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 7: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:57.539379 16306 raft_consensus.cc:686] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:57.562160 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:57.562073 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:57.566284 12102 leader_election.cc:302] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:57.566465 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:57.550485 16427 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Advancing to term 9
W20260808 20:14:57.574388 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 68270us
I20260808 20:14:57.533862 15592 consensus_queue.cc:260] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:57.599433 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:57.615765 12241 raft_consensus.cc:3037] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 2 FOLLOWER]: Advancing to term 3
W20260808 20:14:57.617038 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 50838.1us
I20260808 20:14:57.619671 12249 raft_consensus.cc:1261] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 20:14:57.621284 16306 consensus_queue.cc:237] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:57.624728 12247 raft_consensus.cc:2454] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
W20260808 20:14:57.639247 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 91438.6us
I20260808 20:14:57.609809 16480 leader_election.cc:288] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:57.616801 16501 raft_consensus.cc:2726] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:57.647379 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:57.643565 13014 leader_election.cc:302] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
W20260808 20:14:57.650882 16559 log.cc:924] Time spent T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.150s user 0.000s sys 0.001s
W20260808 20:14:57.653142 12205 leader_election.cc:334] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Timed out: RequestConsensusVote RPC to 127.10.24.1:40747 timed out after 1.544s (SENT)
I20260808 20:14:57.653576 12205 leader_election.cc:302] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:57.657691 16434 raft_consensus.cc:2781] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 20:14:57.658118 16434 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:57.658420 16434 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:14:57.668205 12254 raft_consensus.cc:1226] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:57.669008 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:57.669495 12101 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:57.639402 15592 raft_consensus.cc:3037] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:57.677244 12247 raft_consensus.cc:2379] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 7: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:57.670572 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:14:57.682840 12251 raft_consensus.cc:2454] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:57.695690 12206 leader_election.cc:302] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:57.697929 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:57.699805 13080 raft_consensus.cc:2379] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 8: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:57.701122 12101 leader_election.cc:302] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
W20260808 20:14:57.702400 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 73282.3us
I20260808 20:14:57.702400 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:57.705055 12102 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:57.706099 12101 leader_election.cc:302] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:57.695803 16005 raft_consensus.cc:2781] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:57.694406 16501 raft_consensus.cc:2726] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 20:14:57.694296 16428 raft_consensus.cc:686] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:57.714747 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:57.714416 12243 raft_consensus.cc:2454] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:14:57.718559 12101 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:57.712028 16480 raft_consensus.cc:2781] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:57.724390 16422 raft_consensus.cc:2781] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:14:57.729411 13080 raft_consensus.cc:1226] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 7. Current term is 8. Ops: []
I20260808 20:14:57.730773 13072 raft_consensus.cc:2379] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 7: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
W20260808 20:14:57.731416 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 78409.8us
I20260808 20:14:57.719347 16263 consensus_queue.cc:1035] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.074s
I20260808 20:14:57.715166 16306 raft_consensus.cc:2726] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:57.718048 16005 raft_consensus.cc:686] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:57.721339 16516 raft_consensus.cc:2726] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 20:14:57.742277 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:57.724535 16574 raft_consensus.cc:2741] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260808 20:14:57.731946 16422 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:57.747054 12101 leader_election.cc:302] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:57.751145 16569 raft_consensus.cc:2726] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:57.739640 16480 raft_consensus.cc:686] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:57.747301 16422 raft_consensus.cc:3037] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:14:57.757427 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 76297.3us
I20260808 20:14:57.762100 12101 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:57.765415 16566 raft_consensus.cc:2781] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 20:14:57.767604 16432 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:57.769105 16521 raft_consensus.cc:2781] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:57.770186 16005 consensus_queue.cc:237] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:57.770853 13015 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:57.773082 15931 raft_consensus.cc:2781] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:14:57.773494 15931 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:57.747464 16516 consensus_queue.cc:1046] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260808 20:14:57.773826 15931 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:57.774712 16480 consensus_queue.cc:237] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:57.759446 16542 raft_consensus.cc:2726] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901) with higher term. Message: Invalid argument: T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 7. Current term is 8.
I20260808 20:14:57.780786 12046 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 8.
I20260808 20:14:57.789172 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:57.789392 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:57.790203 13072 raft_consensus.cc:2379] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 8: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:57.759584 16501 raft_consensus.cc:2781] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:14:57.762352 16428 consensus_queue.cc:237] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:57.700753 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:14:57.799260 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:57.796072 13078 raft_consensus.cc:1226] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [term 10 LEADER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 9. Current term is 10. Ops: []
I20260808 20:14:57.812109 13080 raft_consensus.cc:2379] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
W20260808 20:14:57.818164 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 67709.7us
I20260808 20:14:57.818698 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:57.819028 15826 raft_consensus.cc:2726] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:57.820436 12241 raft_consensus.cc:2454] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:14:57.823514 15674 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:57.824419 16432 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:57.825635 15674 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:57.825692 13071 raft_consensus.cc:1226] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6. Ops: [4.5-5.6]
I20260808 20:14:57.827464 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:57.831907 13072 raft_consensus.cc:1226] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 LEADER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 6. Current term is 7. Ops: []
I20260808 20:14:57.832432 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:57.823621 13074 raft_consensus.cc:1226] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:57.830459 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:57.847362 12039 raft_consensus.cc:2354] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 7. Current term is 8.
W20260808 20:14:57.829236 16551 log.cc:924] Time spent T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.431s user 0.002s sys 0.021s
I20260808 20:14:57.829821 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:57.893887 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:57.816193 16501 raft_consensus.cc:686] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:57.897212 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:57.830847 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:57.898391 12206 leader_election.cc:302] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:57.902192 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
W20260808 20:14:57.903168 16575 log.cc:924] Time spent T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260808 20:14:57.911254 13073 raft_consensus.cc:2379] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:57.837270 16521 raft_consensus.cc:686] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:57.822877 16566 raft_consensus.cc:686] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 9 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:57.919083 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:57.830971 16434 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:57.922626 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 156296us
I20260808 20:14:57.830273 16580 consensus_queue.cc:1035] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.088s
W20260808 20:14:57.928030 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 110146us
I20260808 20:14:57.914134 13071 raft_consensus.cc:3037] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:57.931339 16521 consensus_queue.cc:237] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:57.940111 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 93727.5us
I20260808 20:14:57.938490 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:57.949273 12254 raft_consensus.cc:3037] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 6
I20260808 20:14:57.949331 12101 leader_election.cc:302] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:57.929507 16586 raft_consensus.cc:2726] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:57.969529 16501 consensus_queue.cc:237] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:57.972847 16477 raft_consensus.cc:3032] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:57.972718 12206 leader_election.cc:302] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:57.973848 16580 raft_consensus.cc:2726] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:57.973255 16477 raft_consensus.cc:729] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:57.977948 16477 consensus_queue.cc:260] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:57.981639 12206 leader_election.cc:302] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:57.984573 12206 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:57.985535 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:57.986217 12047 raft_consensus.cc:3037] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:57.986758 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:57.987617 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:57.988759 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:57.989243 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:57.989825 12038 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 9.
I20260808 20:14:57.990669 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:57.991120 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:57.991813 12040 raft_consensus.cc:2379] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 4: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:57.994095 12047 raft_consensus.cc:2454] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:14:57.994967 15592 raft_consensus.cc:2726] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 20:14:57.987129 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:57.997764 12206 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:57.998517 12039 raft_consensus.cc:2379] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 6: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
W20260808 20:14:58.004735 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 55488.9us
W20260808 20:14:57.973747 16562 log.cc:924] Time spent T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.505s user 0.002s sys 0.000s
I20260808 20:14:58.011782 12254 raft_consensus.cc:2454] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:14:58.015950 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:58.023420 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:58.015592 15592 raft_consensus.cc:2781] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader election won for term 8
I20260808 20:14:58.046504 13015 leader_election.cc:398] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 election: Vote denied by peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747) with higher term. Message: Invalid argument: T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 4.
I20260808 20:14:58.048601 13015 leader_election.cc:401] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260808 20:14:58.044579 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:58.054035 12036 raft_consensus.cc:2379] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 3: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:58.046360 12101 leader_election.cc:302] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:58.057330 12047 raft_consensus.cc:1226] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 6. Current term is 7. Ops: []
I20260808 20:14:57.924855 16422 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:58.043049 16542 raft_consensus.cc:2781] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 20:14:58.048229 16586 raft_consensus.cc:2741] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260808 20:14:58.082031 15592 raft_consensus.cc:686] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 8 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
W20260808 20:14:58.093688 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 94003.5us
I20260808 20:14:58.063454 15931 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:58.117556 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:58.118184 12040 raft_consensus.cc:3037] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:58.092717 16434 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 9 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:14:58.124622 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 78347.6us
I20260808 20:14:58.097957 16542 raft_consensus.cc:686] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:58.127599 13078 raft_consensus.cc:1226] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 7. Current term is 8. Ops: []
I20260808 20:14:58.112542 16240 raft_consensus.cc:2726] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747) with higher term. Message: Invalid argument: T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 4.
W20260808 20:14:58.127549 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 60800.1us
I20260808 20:14:58.065493 16566 consensus_queue.cc:237] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:58.116348 16511 raft_consensus.cc:2781] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:14:58.128976 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:58.130262 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:58.139497 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:58.140559 12048 raft_consensus.cc:2396] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260808 20:14:58.140010 16542 consensus_queue.cc:237] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:58.141579 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:58.142433 12048 raft_consensus.cc:2396] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 6 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260808 20:14:58.143158 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:58.143739 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:58.143868 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.144028 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:58.144659 12038 raft_consensus.cc:2454] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:58.145591 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:58.146401 12039 raft_consensus.cc:2396] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260808 20:14:58.146627 16597 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:58.147307 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:58.147135 16597 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:58.148399 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:58.149246 12041 raft_consensus.cc:2396] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260808 20:14:58.151355 16595 log.cc:924] Time spent T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.183s user 0.002s sys 0.000s
I20260808 20:14:58.152614 16422 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:58.151242 15931 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:58.124401 15592 consensus_queue.cc:237] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:58.163897 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:58.144579 12146 raft_consensus.cc:2379] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 6: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
W20260808 20:14:58.134397 12205 leader_election.cc:334] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Timed out: RequestConsensusVote RPC to 127.10.24.1:40747 timed out after 1.513s (SENT)
I20260808 20:14:58.144572 12137 raft_consensus.cc:2379] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 7: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:58.166553 12205 leader_election.cc:302] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:58.168448 13071 raft_consensus.cc:2454] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:14:58.143764 12048 raft_consensus.cc:2454] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 7.
I20260808 20:14:58.178870 11994 leader_election.cc:302] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:58.181317 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:58.182873 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 58814us
I20260808 20:14:58.183670 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:58.185249 12046 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:58.191934 12041 raft_consensus.cc:2379] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 8: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:58.191844 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:14:58.185490 12040 raft_consensus.cc:2454] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 7.
I20260808 20:14:58.191970 13075 raft_consensus.cc:1226] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 LEADER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 7. Current term is 8. Ops: []
I20260808 20:14:58.195832 13015 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:58.194471 12048 raft_consensus.cc:2454] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:14:58.191886 12036 raft_consensus.cc:2396] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 9 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260808 20:14:58.199183 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:58.199175 13071 raft_consensus.cc:1226] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:58.200448 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:14:58.196722 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 62353.7us
I20260808 20:14:58.212065 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:58.213027 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:58.213671 12146 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8.
I20260808 20:14:58.213927 12249 raft_consensus.cc:1226] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 7. Current term is 8. Ops: []
W20260808 20:14:58.193816 16587 log.cc:924] Time spent T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.192s user 0.005s sys 0.000s
I20260808 20:14:58.215155 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:58.221724 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:58.182610 16511 raft_consensus.cc:686] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:58.229159 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:58.185297 16434 raft_consensus.cc:2726] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:58.228024 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:58.229895 12036 raft_consensus.cc:2379] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 5: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:58.230079 12150 raft_consensus.cc:2454] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 7.
I20260808 20:14:58.230553 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:58.235014 12146 raft_consensus.cc:1226] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7. Ops: [4.4-6.5]
W20260808 20:14:58.236186 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 57388.1us
I20260808 20:14:58.237541 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:58.238440 12101 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:58.241745 12036 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 7.
W20260808 20:14:58.242376 12205 leader_election.cc:334] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Timed out: RequestConsensusVote RPC to 127.10.24.1:40747 timed out after 1.519s (SENT)
I20260808 20:14:58.242802 12205 leader_election.cc:302] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:58.243834 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:58.242230 12248 raft_consensus.cc:1261] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260808 20:14:58.230427 15931 raft_consensus.cc:2726] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:58.243863 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:58.229736 16508 raft_consensus.cc:2726] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:58.245857 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:58.229501 16306 raft_consensus.cc:482] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:14:58.248606 15699 raft_consensus.cc:2726] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 20:14:58.249079 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.262131 12206 leader_election.cc:302] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:58.262339 12146 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:14:58.247983 16566 consensus_queue.cc:1046] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.015s
I20260808 20:14:58.263279 16609 consensus_queue.cc:1046] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 20:14:58.263898 16609 consensus_queue.cc:1225] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer e5bd9a4616ce4acfa20dc7a002984a20 log is divergent from this leader: its last log entry 8.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260808 20:14:58.248505 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.265656 12140 raft_consensus.cc:2379] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 7: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:58.231117 16597 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:58.266618 12040 raft_consensus.cc:2454] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:14:58.266749 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:58.214748 15826 consensus_queue.cc:1046] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
I20260808 20:14:58.251691 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:58.252272 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:58.252004 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:58.260164 11994 leader_election.cc:302] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:58.275135 12247 raft_consensus.cc:2379] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:58.247745 12046 raft_consensus.cc:2454] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 7.
I20260808 20:14:58.275056 12251 raft_consensus.cc:2379] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 8: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:58.276060 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:58.276217 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:58.277168 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:58.276659 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.277947 12241 raft_consensus.cc:2379] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:58.276553 16597 raft_consensus.cc:2726] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:58.278928 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:58.279510 12247 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 9.
I20260808 20:14:58.280297 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:58.248086 11896 catalog_manager.cc:5495] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 2 to 5, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4). New cstate: current_term: 5 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:14:58.276791 16434 raft_consensus.cc:2726] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:58.281605 16511 consensus_queue.cc:237] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:58.282291 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:58.252377 16306 raft_consensus.cc:504] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:58.283372 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:58.283931 12247 raft_consensus.cc:2362] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7. Re-sending same reply.
I20260808 20:14:58.284739 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:58.282636 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:58.285389 12247 raft_consensus.cc:2354] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 6. Current term is 7.
I20260808 20:14:58.281306 12244 raft_consensus.cc:1261] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 20:14:58.285771 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:58.285800 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:58.283694 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:58.286936 12046 raft_consensus.cc:2379] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 3: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:58.276322 11994 leader_election.cc:302] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:58.291196 12248 raft_consensus.cc:2396] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260808 20:14:58.291566 12251 raft_consensus.cc:2379] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 8: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:58.250113 16477 raft_consensus.cc:3037] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:58.297132 13080 raft_consensus.cc:1261] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 20:14:58.293875 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:58.297096 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:58.298369 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:58.299489 12146 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 9.
I20260808 20:14:58.296243 12150 raft_consensus.cc:1226] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 7. Current term is 8. Ops: []
W20260808 20:14:58.294344 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 110789us
I20260808 20:14:58.296103 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:58.307227 12245 raft_consensus.cc:2379] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:58.308338 12248 raft_consensus.cc:1203] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 10 FOLLOWER]: Deduplicated request from leader. Original: 6.7->[7.8-9.9] Dedup: 9.9->[]
I20260808 20:14:58.308451 12245 raft_consensus.cc:1261] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260808 20:14:58.308861 12248 raft_consensus.cc:1226] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 10 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 9. Current term is 10. Ops: [7.8-9.9]
I20260808 20:14:58.311173 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:58.308851 12245 pending_rounds.cc:76] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a: Aborting all ops after (but not including) 4
W20260808 20:14:58.312350 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 82959.1us
I20260808 20:14:58.322747 12243 raft_consensus.cc:1226] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 8. Current term is 9. Ops: [5.3-8.5]
I20260808 20:14:58.313509 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:58.321051 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:58.324503 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:58.325372 12047 raft_consensus.cc:2379] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 8: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:58.325663 12248 raft_consensus.cc:1261] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 20:14:58.327708 12241 raft_consensus.cc:1226] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 6. Current term is 7. Ops: []
I20260808 20:14:58.329031 12241 raft_consensus.cc:1261] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 20:14:58.320472 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:58.312304 12245 pending_rounds.cc:97] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260808 20:14:58.331634 12245 raft_consensus.cc:2866] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:58.295717 16542 raft_consensus.cc:3032] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 20:14:58.332866 12244 raft_consensus.cc:2379] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 3: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:58.336723 12040 raft_consensus.cc:1261] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260808 20:14:58.337881 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:58.338390 12040 raft_consensus.cc:3037] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:58.339898 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:58.324513 16597 raft_consensus.cc:2726] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 20:14:58.341382 16617 log.cc:924] Time spent T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.114s user 0.000s sys 0.001s
I20260808 20:14:58.341897 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:58.341241 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:58.343214 12043 raft_consensus.cc:2396] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260808 20:14:58.341900 12041 raft_consensus.cc:1226] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7. Ops: [3.3-6.5]
I20260808 20:14:58.341003 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:58.343452 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.342239 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:58.344861 12045 raft_consensus.cc:3037] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Advancing to term 9
I20260808 20:14:58.345961 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:58.346101 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:58.347568 12150 raft_consensus.cc:2379] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 7: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:58.348726 13080 raft_consensus.cc:1226] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:58.349457 12046 raft_consensus.cc:2379] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 7: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:58.326114 15592 raft_consensus.cc:3037] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
W20260808 20:14:58.350520 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 88485.5us
I20260808 20:14:58.334662 16542 raft_consensus.cc:729] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:58.351168 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:58.354895 12001 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:58.356614 12040 raft_consensus.cc:2454] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:14:58.352110 12206 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:58.357673 12048 raft_consensus.cc:3032] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 20:14:58.298406 15908 consensus_queue.cc:1035] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20260808 20:14:58.358045 12048 raft_consensus.cc:729] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
W20260808 20:14:58.365095 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 88861us
I20260808 20:14:58.365442 16508 raft_consensus.cc:2726] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 20:14:58.363085 12140 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
I20260808 20:14:58.366196 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.368783 12206 leader_election.cc:302] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:58.370154 12146 raft_consensus.cc:2354] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 6. Current term is 7.
I20260808 20:14:58.310663 15674 consensus_queue.cc:1035] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260808 20:14:58.322036 16501 consensus_queue.cc:1035] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.032s
I20260808 20:14:58.371236 12048 consensus_queue.cc:260] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:58.376263 12048 raft_consensus.cc:3037] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:58.367925 16434 raft_consensus.cc:2781] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:14:58.341029 15699 consensus_queue.cc:1035] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.030s
I20260808 20:14:58.382768 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.383335 12146 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:58.379844 16574 raft_consensus.cc:2726] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:58.386363 16566 raft_consensus.cc:3032] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 20:14:58.386754 16566 raft_consensus.cc:729] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:58.387486 16566 consensus_queue.cc:260] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:58.379350 16434 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:58.389511 16434 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:58.392526 13080 raft_consensus.cc:1261] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260808 20:14:58.394203 12048 raft_consensus.cc:1261] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 20:14:58.395321 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:58.395653 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:58.396339 12254 raft_consensus.cc:2354] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5.
I20260808 20:14:58.400235 12249 raft_consensus.cc:2354] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 6. Current term is 7.
I20260808 20:14:58.388752 16566 raft_consensus.cc:3037] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:58.410065 12206 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:58.407229 12146 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8.
I20260808 20:14:58.411417 16597 consensus_queue.cc:1046] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 20:14:58.412664 16508 raft_consensus.cc:3032] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 20:14:58.413056 16508 raft_consensus.cc:729] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:58.413949 16508 consensus_queue.cc:260] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:58.417233 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.418412 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.419194 12150 raft_consensus.cc:2379] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 8: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:14:58.420971 16586 consensus_queue.cc:1046] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:14:58.421209 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.421670 16586 consensus_queue.cc:1225] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer 78b9263889cd4f1582578a3aa4f644d9 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260808 20:14:58.422158 16586 raft_consensus.cc:3032] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Stepping down as leader of term 6
I20260808 20:14:58.422456 16586 raft_consensus.cc:729] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:58.422948 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.423216 16586 consensus_queue.cc:260] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:58.424369 16586 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:58.424490 15592 raft_consensus.cc:2726] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:58.379105 16306 leader_election.cc:288] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:14:58.431548 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 119619us
W20260808 20:14:58.435271 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 55611.4us
W20260808 20:14:58.435855 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 53575.3us
I20260808 20:14:58.437997 12245 raft_consensus.cc:1261] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260808 20:14:58.447430 11994 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:58.457432 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
W20260808 20:14:58.457789 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 125284us
I20260808 20:14:58.463629 12243 raft_consensus.cc:1226] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:58.460799 16574 raft_consensus.cc:2781] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader election won for term 9
I20260808 20:14:58.469041 16574 raft_consensus.cc:686] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [term 9 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:58.469889 16574 consensus_queue.cc:237] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:58.471477 16601 raft_consensus.cc:2726] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:58.472779 11994 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:58.473323 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:58.484192 12045 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 9.
I20260808 20:14:58.489241 12001 leader_election.cc:302] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:58.484545 16601 raft_consensus.cc:2781] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 20:14:58.491459 16601 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:58.492528 11994 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:58.404990 16501 consensus_queue.cc:1035] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.074s
I20260808 20:14:58.478102 16609 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:58.503106 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.503901 12146 raft_consensus.cc:2379] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:14:58.505775 12001 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:58.505493 13015 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:58.511376 11994 leader_election.cc:398] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Vote denied by peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) with higher term. Message: Invalid argument: T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 6. Current term is 7.
I20260808 20:14:58.513079 16388 consensus_queue.cc:1035] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
I20260808 20:14:58.514077 16438 raft_consensus.cc:2726] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:58.405189 16542 consensus_queue.cc:260] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:58.517577 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:58.494858 16601 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 10 FOLLOWER]: Advancing to term 11
I20260808 20:14:58.499495 16599 raft_consensus.cc:2726] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:58.539166 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.538456 15666 raft_consensus.cc:2726] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:58.547109 11994 leader_election.cc:302] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:58.524308 16427 raft_consensus.cc:2726] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 20:14:58.549023 13015 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
W20260808 20:14:58.549650 12205 leader_election.cc:334] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747): Timed out: RequestConsensusVote RPC to 127.10.24.1:40747 timed out after 1.529s (SENT)
I20260808 20:14:58.551358 12205 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:58.548869 16521 raft_consensus.cc:482] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:58.537544 16438 consensus_queue.cc:1046] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.018s
I20260808 20:14:58.559516 13075 raft_consensus.cc:1261] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 20:14:58.559659 12146 raft_consensus.cc:1226] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5. Ops: []
I20260808 20:14:58.562134 12205 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:58.560534 16599 raft_consensus.cc:2726] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:58.569531 16599 consensus_queue.cc:1046] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 20:14:58.570268 16400 raft_consensus.cc:2781] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 20:14:58.550424 15931 consensus_queue.cc:1035] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
W20260808 20:14:58.572592 16620 log.cc:924] Time spent T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.218s user 0.003s sys 0.000s
I20260808 20:14:58.517343 16609 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:58.609228 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 44592.3us
W20260808 20:14:58.611950 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 50311.1us
I20260808 20:14:58.616578 12141 raft_consensus.cc:1261] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 20:14:58.557372 16580 raft_consensus.cc:2726] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260808 20:14:58.640681 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 81362.4us
I20260808 20:14:58.642688 12001 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:58.613464 16400 raft_consensus.cc:686] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:14:58.644357 12146 raft_consensus.cc:1226] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:58.557972 16427 consensus_queue.cc:1046] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.081s
W20260808 20:14:58.649727 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 87666.1us
I20260808 20:14:58.655524 12205 leader_election.cc:302] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:14:58.656584 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 45736.6us
I20260808 20:14:58.648639 15592 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Advancing to term 10
I20260808 20:14:58.575512 16521 raft_consensus.cc:504] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:58.668011 12205 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:58.665067 12140 raft_consensus.cc:1226] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [term 9 LEADER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 8. Current term is 9. Ops: []
I20260808 20:14:58.613750 16574 consensus_queue.cc:1035] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260808 20:14:58.678758 12146 raft_consensus.cc:1226] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:58.679872 16306 raft_consensus.cc:2781] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:14:58.680307 16306 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:58.680619 16306 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:58.681097 16609 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:58.667616 16438 consensus_queue.cc:1225] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer e5bd9a4616ce4acfa20dc7a002984a20 log is divergent from this leader: its last log entry 10.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260808 20:14:58.682922 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:58.685971 12254 raft_consensus.cc:1261] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 20:14:58.687753 16306 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:58.687914 15699 consensus_queue.cc:1035] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:58.690536 16521 leader_election.cc:288] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:58.691322 12144 raft_consensus.cc:1226] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260808 20:14:58.694954 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:58.696645 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:58.698351 16438 raft_consensus.cc:3032] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 20:14:58.698786 16438 raft_consensus.cc:729] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:58.699717 16438 consensus_queue.cc:260] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:58.701061 12140 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 10.
I20260808 20:14:58.702837 16508 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:58.702996 16306 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:58.703452 16400 consensus_queue.cc:237] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:58.706287 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.704631 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.661576 16580 raft_consensus.cc:2726] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:58.721177 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
W20260808 20:14:58.698889 16621 log.cc:924] Time spent T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.242s user 0.002s sys 0.000s
W20260808 20:14:58.734062 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 76521.8us
I20260808 20:14:58.718528 16644 raft_consensus.cc:3032] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:14:58.732991 11896 catalog_manager.cc:5495] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 4 to 6, leader changed from e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 6 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:58.709863 16597 raft_consensus.cc:2726] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 20:14:58.761322 12144 raft_consensus.cc:1261] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 20:14:58.761219 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.744113 12140 raft_consensus.cc:2362] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8. Re-sending same reply.
I20260808 20:14:58.762315 11994 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:58.767266 12146 raft_consensus.cc:2379] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 8: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:58.705302 16438 raft_consensus.cc:3037] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Advancing to term 10
I20260808 20:14:58.770525 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.771248 12146 raft_consensus.cc:2379] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 5: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:58.772046 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
W20260808 20:14:58.774093 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 114361us
I20260808 20:14:58.775100 12205 leader_election.cc:302] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:58.776320 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.778858 12141 raft_consensus.cc:2379] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 8: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:58.744542 16644 raft_consensus.cc:729] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:58.796968 12146 raft_consensus.cc:1226] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:58.799634 16542 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Advancing to term 8
W20260808 20:14:58.805944 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 43710.2us
I20260808 20:14:58.808324 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.809166 12144 raft_consensus.cc:2379] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 7: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:58.813418 11994 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:58.816643 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.817082 12250 raft_consensus.cc:1261] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260808 20:14:58.782111 16597 raft_consensus.cc:2726] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:58.818960 12245 raft_consensus.cc:1261] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 20:14:58.815346 16599 consensus_queue.cc:1046] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
W20260808 20:14:58.825368 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 50350.2us
I20260808 20:14:58.826176 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.827323 12205 leader_election.cc:302] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:58.807945 15699 raft_consensus.cc:482] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:14:58.828550 12205 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:58.835465 16306 raft_consensus.cc:2726] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:58.824815 11894 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:57500 (request call id 313) took 11002 ms. Trace:
I20260808 20:14:58.837689 12141 raft_consensus.cc:1226] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:58.838224 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:58.839113 12146 raft_consensus.cc:2454] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:58.825714 16599 consensus_queue.cc:1225] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer b9d94f35ea15421aa1d721201c782d8a log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20260808 20:14:58.842417 16642 log.cc:924] Time spent T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.197s user 0.002s sys 0.000s
I20260808 20:14:58.837349 16597 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:58.843051 12140 raft_consensus.cc:1261] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 20:14:58.843189 16597 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:58.843052 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.810036 16580 raft_consensus.cc:2726] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:58.843937 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
W20260808 20:14:58.834187 16626 log.cc:924] Time spent T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.272s user 0.001s sys 0.000s
I20260808 20:14:58.843958 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:58.859397 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:58.861462 13078 raft_consensus.cc:1261] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 20:14:58.834339 16574 raft_consensus.cc:2726] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 20:14:58.903194 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.904546 13072 raft_consensus.cc:1226] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 7. Current term is 8. Ops: []
I20260808 20:14:58.906108 12141 raft_consensus.cc:1261] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 10 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 10 index: 12. (index mismatch)
I20260808 20:14:58.839891 16588 consensus_queue.cc:1035] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.021s
I20260808 20:14:58.859428 16427 raft_consensus.cc:2726] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:58.909927 12105 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:14:58.912878 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 270260us
I20260808 20:14:58.915592 12105 leader_election.cc:302] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:58.834381 15674 consensus_queue.cc:1035] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.075s
I20260808 20:14:58.909307 11896 catalog_manager.cc:5495] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 5 to 6, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 6 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:58.901261 16599 raft_consensus.cc:3032] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 20:14:58.916846 12001 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:58.922421 12144 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 8.
I20260808 20:14:58.836700 15699 raft_consensus.cc:504] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:58.944280 16647 raft_consensus.cc:3032] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:58.944684 16647 raft_consensus.cc:729] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:58.945500 16647 consensus_queue.cc:260] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:58.950987 12147 raft_consensus.cc:2454] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:14:58.953567 12205 leader_election.cc:302] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:58.934788 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
W20260808 20:14:58.952693 16628 log.cc:924] Time spent T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.391s user 0.000s sys 0.002s
I20260808 20:14:58.921320 16588 consensus_queue.cc:1225] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer b9d94f35ea15421aa1d721201c782d8a log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260808 20:14:58.863302 16633 consensus_queue.cc:1035] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.068s
I20260808 20:14:58.934437 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.962555 13080 raft_consensus.cc:1261] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 20:14:58.966297 12144 raft_consensus.cc:2379] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 7: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:58.967298 12142 raft_consensus.cc:1261] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260808 20:14:58.909029 16609 consensus_queue.cc:1035] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.050s
I20260808 20:14:58.930133 16599 raft_consensus.cc:729] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:58.977284 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.984368 12147 raft_consensus.cc:2379] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 8: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:14:58.958020 16477 raft_consensus.cc:2726] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:14:58.996073 12150 raft_consensus.cc:1226] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 6. Current term is 7. Ops: []
I20260808 20:14:59.006067 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:59.007278 12144 raft_consensus.cc:1261] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260808 20:14:59.000756 12048 raft_consensus.cc:3032] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:59.007735 12048 raft_consensus.cc:729] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:59.010143 12045 raft_consensus.cc:1261] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 20:14:58.975793 16574 raft_consensus.cc:2726] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:59.017891 12250 raft_consensus.cc:1261] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 20:14:59.014557 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:59.019026 12039 raft_consensus.cc:3037] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:14:58.836426 11894 rpcz_store.cc:276] 0808 20:14:47.822299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:47.822493 (+ 194us) service_pool.cc:227] Handling call
0808 20:14:47.995426 (+172933us) spinlock_profiling.cc:243] Waited 26.4 ms on lock 0x7b5400002dd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825faf0 00007fa9682e2f2a 00007fa96e234d6c 00007fa96e32ec3e 00007fa96c640fb4 00007fa96c640f42 00007fa96c640e71 00007fa96c640dec 00007fa96c6400a2 00007fa96670494c 00007fa966703d86
0808 20:14:58.824568 (+10829142us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 20:14:56.617267 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 20:14:56.638192 (+ 20925us) write_op.cc:432] Acquiring schema lock in shared mode
0808 20:14:56.638226 (+ 34us) write_op.cc:435] Acquired schema lock
0808 20:14:56.638237 (+ 11us) tablet.cc:660] Decoding operations
0808 20:14:56.691010 (+ 52773us) write_op.cc:620] Acquiring the partition lock for write op
0808 20:14:56.691072 (+ 62us) write_op.cc:641] Partition lock acquired for write op
0808 20:14:56.691098 (+ 26us) tablet.cc:683] Acquiring locks for 22 operations
0808 20:14:56.692565 (+ 1467us) tablet.cc:699] Row locks acquired
0808 20:14:56.692586 (+ 21us) write_op.cc:260] PREPARE: finished
0808 20:14:56.692747 (+ 161us) write_op.cc:270] Start()
0808 20:14:56.692848 (+ 101us) write_op.cc:276] Timestamp: P: 1786220096692696 usec, L: 0
0808 20:14:56.692868 (+ 20us) op_driver.cc:331] REPLICATION: starting
0808 20:14:56.831279 (+138411us) log.cc:842] Serialized 8616 byte log entry
0808 20:14:57.089723 (+258444us) op_driver.cc:444] REPLICATION: finished
0808 20:14:57.273476 (+183753us) write_op.cc:301] APPLY: starting
0808 20:14:57.273603 (+ 127us) tablet.cc:1365] starting BulkCheckPresence
0808 20:14:57.274099 (+ 496us) tablet.cc:1368] finished BulkCheckPresence
0808 20:14:57.292987 (+ 18888us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 20:14:57.328550 (+ 35563us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 20:14:57.328961 (+ 411us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=22
0808 20:14:57.329011 (+ 50us) write_op.cc:312] APPLY: finished
0808 20:14:57.329561 (+ 550us) log.cc:842] Serialized 152 byte log entry
0808 20:14:57.347881 (+ 18320us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:14:57.350074 (+ 2193us) write_op.cc:454] Released schema lock
0808 20:14:57.350160 (+ 86us) write_op.cc:341] FINISH: Updating metrics
0808 20:14:57.350626 (+ 466us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:14:57.350653 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4659,"reported_tablets":40,"spinlock_wait_cycles":60629888,"child_traces":[["op",{"apply.queue_time_us":162662,"apply.run_cpu_time_us":10303,"apply.run_wall_time_us":79953,"num_ops":22,"prepare.queue_time_us":577560,"prepare.run_cpu_time_us":18535,"prepare.run_wall_time_us":403449,"raft.queue_time_us":93391,"raft.run_cpu_time_us":47975,"raft.run_wall_time_us":95864,"replication_time_us":395845,"thread_start_us":378285,"threads_started":4,"wal-append.queue_time_us":157814,"wal-append.run_cpu_time_us":16949,"wal-append.run_wall_time_us":1341760}]]}
I20260808 20:14:58.933610 16601 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:59.006570 11896 catalog_manager.cc:5495] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } }
I20260808 20:14:58.960798 15931 consensus_queue.cc:1035] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.051s
W20260808 20:14:59.021967 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 68471.9us
I20260808 20:14:59.027974 12205 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:59.022783 12048 consensus_queue.cc:260] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:59.029155 12048 raft_consensus.cc:3037] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:59.024471 12041 raft_consensus.cc:1261] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 20:14:59.015143 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:58.904877 16644 consensus_queue.cc:260] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:59.040478 12048 raft_consensus.cc:1261] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 20:14:59.032308 12045 raft_consensus.cc:1261] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 20:14:59.041590 12205 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:59.031615 12138 raft_consensus.cc:2362] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 8. Re-sending same reply.
I20260808 20:14:59.025451 12038 raft_consensus.cc:1261] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 20:14:59.042269 16574 raft_consensus.cc:2726] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 20:14:59.042795 11896 catalog_manager.cc:5495] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } }
I20260808 20:14:59.048286 12205 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:59.053378 12048 raft_consensus.cc:1261] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 20:14:59.054643 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:59.054172 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:59.060971 12036 raft_consensus.cc:2396] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 9 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260808 20:14:59.061506 12048 raft_consensus.cc:2379] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 8: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:14:59.062438 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:59.063030 12048 raft_consensus.cc:2362] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6. Re-sending same reply.
I20260808 20:14:59.047201 16647 raft_consensus.cc:3037] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
W20260808 20:14:59.106451 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 299390us
I20260808 20:14:59.107575 12045 raft_consensus.cc:1226] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7. Ops: []
I20260808 20:14:59.018695 16306 consensus_queue.cc:1035] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20260808 20:14:59.057569 16511 raft_consensus.cc:482] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:59.108145 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:59.122272 12041 raft_consensus.cc:2454] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:59.123122 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:59.057104 16434 consensus_meta.cc:287] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.666s user 0.008s sys 0.003s
I20260808 20:14:59.107378 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:14:59.109241 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 97654.4us
I20260808 20:14:59.109549 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:14:59.042979 16427 raft_consensus.cc:482] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:59.133200 11994 leader_election.cc:302] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:59.068832 16580 raft_consensus.cc:2726] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 20:14:59.064584 16633 raft_consensus.cc:2726] T 2a41c716cfc94652a2fc6957a1bd2a78 P b9d94f35ea15421aa1d721201c782d8a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 20:14:59.137384 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:59.138419 12045 raft_consensus.cc:1261] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260808 20:14:59.129599 12000 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:59.132942 16427 raft_consensus.cc:504] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:59.133507 16511 raft_consensus.cc:504] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:59.143009 16427 leader_election.cc:288] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:59.143405 16511 leader_election.cc:288] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:59.055037 12250 pending_rounds.cc:76] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a: Aborting all ops after (but not including) 2
I20260808 20:14:59.127744 12047 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:59.043284 16521 consensus_queue.cc:1035] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.075s
I20260808 20:14:59.125545 11896 catalog_manager.cc:5495] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 4 to 5, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 5 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:59.131353 12102 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:59.146555 12250 pending_rounds.cc:97] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a: Aborting uncommitted NO_OP operation due to leader change: 6.3
I20260808 20:14:59.152971 12250 raft_consensus.cc:2866] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:14:59.126992 16597 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:59.151607 11896 catalog_manager.cc:5495] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 6 to 7, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 7 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:59.131233 16605 raft_consensus.cc:2726] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:59.159778 12000 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:59.111395 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:59.160060 12039 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:14:59.162822 11896 catalog_manager.cc:5495] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 4 to 7, leader changed from e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 7 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:59.129410 16434 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:59.164801 11896 catalog_manager.cc:5495] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 3 to 4, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to b9d94f35ea15421aa1d721201c782d8a (127.10.24.3). New cstate: current_term: 4 leader_uuid: "b9d94f35ea15421aa1d721201c782d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } }
I20260808 20:14:59.166190 11896 catalog_manager.cc:5495] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 1 to 2, leader changed from 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:59.166906 16434 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:14:59.167629 11896 catalog_manager.cc:5495] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 reported cstate change: term changed from 1 to 2, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 2 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:14:59.168603 16663 raft_consensus.cc:2726] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:59.048849 16599 consensus_queue.cc:260] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:59.168622 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:14:59.173897 12045 raft_consensus.cc:1226] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 4. Ops: []
I20260808 20:14:59.161001 16597 raft_consensus.cc:2726] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 20:14:59.181556 12000 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:59.182740 12000 leader_election.cc:302] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:59.183918 12000 leader_election.cc:302] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:59.184819 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:59.177357 12102 leader_election.cc:398] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 election: Vote denied by peer 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747) with higher term. Message: Invalid argument: T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 7. Current term is 8.
I20260808 20:14:59.159020 15592 raft_consensus.cc:2726] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 20:14:59.202286 12102 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:59.215047 12047 raft_consensus.cc:2454] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 8.
I20260808 20:14:59.170557 15674 raft_consensus.cc:2726] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:14:59.185137 16663 raft_consensus.cc:2726] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260808 20:14:59.220763 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 87627.8us
I20260808 20:14:59.216440 12039 raft_consensus.cc:1226] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
W20260808 20:14:59.217510 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 169284us
I20260808 20:14:59.185567 16597 raft_consensus.cc:2726] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 20:14:59.223284 12142 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:59.218761 12102 leader_election.cc:302] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:59.224107 12205 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:59.193492 15699 leader_election.cc:288] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:59.228577 12102 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:59.230556 12205 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:59.222632 16599 raft_consensus.cc:3037] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Advancing to term 9
W20260808 20:14:59.227746 16586 consensus_meta.cc:287] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.803s user 0.006s sys 0.007s
I20260808 20:14:59.242923 12102 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:59.234002 16306 raft_consensus.cc:2726] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 20:14:59.244665 12205 leader_election.cc:302] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:59.233386 16601 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 11 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:59.246340 11994 leader_election.cc:398] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Vote denied by peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) with higher term. Message: Invalid argument: T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5.
I20260808 20:14:59.246464 12205 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:59.226620 15592 consensus_queue.cc:1035] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260808 20:14:59.247428 11994 leader_election.cc:401] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260808 20:14:59.238852 16574 raft_consensus.cc:2781] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:59.251461 12102 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:14:59.285213 11994 leader_election.cc:398] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Vote denied by peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) with higher term. Message: Invalid argument: T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 6. Current term is 7.
I20260808 20:14:59.286198 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:59.287788 11994 leader_election.cc:401] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260808 20:14:59.236972 16597 consensus_queue.cc:1035] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260808 20:14:59.234711 16434 consensus_queue.cc:1046] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260808 20:14:59.286278 16670 raft_consensus.cc:2741] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260808 20:14:59.235764 16663 consensus_queue.cc:1046] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
W20260808 20:14:59.289700 16668 log.cc:924] Time spent T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.128s user 0.000s sys 0.001s
I20260808 20:14:59.293054 16681 raft_consensus.cc:2726] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 20:14:59.293514 16677 raft_consensus.cc:2741] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
W20260808 20:14:59.293926 16657 log.cc:924] Time spent T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.156s user 0.000s sys 0.000s
W20260808 20:14:59.294256 16665 log.cc:924] Time spent T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.177s user 0.000s sys 0.000s
I20260808 20:14:59.294935 12142 raft_consensus.cc:2454] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 8.
I20260808 20:14:59.254953 16671 raft_consensus.cc:2726] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:14:59.296089 12102 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:59.255016 16306 raft_consensus.cc:2781] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader election won for term 8
I20260808 20:14:59.253468 16675 raft_consensus.cc:2726] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:14:59.247072 16521 raft_consensus.cc:2726] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260808 20:14:59.286314 16666 log.cc:924] Time spent T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.186s user 0.000s sys 0.001s
I20260808 20:14:59.255200 16574 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
W20260808 20:14:59.307808 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 120540us
W20260808 20:14:59.286504 16652 log.cc:924] Time spent T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.262s user 0.001s sys 0.000s
I20260808 20:14:59.253545 16674 raft_consensus.cc:2726] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:14:59.288326 15674 raft_consensus.cc:2726] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 20:14:59.247045 16586 consensus_queue.cc:1035] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.059s
I20260808 20:14:59.296126 16601 raft_consensus.cc:2726] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) with higher term. Message: Invalid argument: T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 4. Current term is 5.
I20260808 20:14:59.354696 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:59.254911 16664 consensus_queue.cc:1035] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.178s
I20260808 20:14:59.354349 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:59.359362 16684 raft_consensus.cc:2726] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 20:14:59.359758 16692 consensus_queue.cc:1046] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.046s
I20260808 20:14:59.359429 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:14:59.308032 16592 consensus_queue.cc:1046] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20260808 20:14:59.413892 16592 raft_consensus.cc:3032] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 20:14:59.414270 16592 raft_consensus.cc:729] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:59.415139 16592 consensus_queue.cc:260] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:59.360690 16701 raft_consensus.cc:2726] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) with higher term. Message: Invalid argument: T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 6. Current term is 7.
I20260808 20:14:59.375448 12000 leader_election.cc:302] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:14:59.418951 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 43569.8us
I20260808 20:14:59.377650 12141 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 8.
I20260808 20:14:59.325328 16633 raft_consensus.cc:482] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:59.421277 16663 consensus_queue.cc:1046] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 20:14:59.421370 16633 raft_consensus.cc:504] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:59.422856 16601 consensus_queue.cc:1035] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:14:59.423609 16633 leader_election.cc:288] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:14:59.384542 12146 raft_consensus.cc:1261] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260808 20:14:59.424206 16566 consensus_meta.cc:287] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.981s user 0.010s sys 0.001s
I20260808 20:14:59.351193 16306 raft_consensus.cc:686] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 8 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:59.426234 16306 consensus_queue.cc:237] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:14:59.356501 16677 raft_consensus.cc:2781] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 20:14:59.429958 16677 raft_consensus.cc:686] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:59.430229 16675 consensus_queue.cc:1035] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 20:14:59.430612 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:14:59.430790 16677 consensus_queue.cc:237] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:59.430279 16701 raft_consensus.cc:3032] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:14:59.432085 16701 raft_consensus.cc:729] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:59.432934 16701 consensus_queue.cc:260] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:59.433921 16655 log.cc:924] Time spent T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.453s user 0.004s sys 0.000s
I20260808 20:14:59.434345 16701 raft_consensus.cc:3037] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:59.383984 12241 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8.
I20260808 20:14:59.358779 16671 consensus_queue.cc:1035] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260808 20:14:59.356099 16574 raft_consensus.cc:3037] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:14:59.416517 16592 raft_consensus.cc:3037] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:14:59.441251 11994 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:59.395473 12043 raft_consensus.cc:1261] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260808 20:14:59.442098 16670 raft_consensus.cc:2781] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 20:14:59.442370 12043 pending_rounds.cc:76] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting all ops after (but not including) 5
W20260808 20:14:59.395659 16508 consensus_meta.cc:287] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.688s user 0.006s sys 0.004s
I20260808 20:14:59.442651 12043 pending_rounds.cc:97] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260808 20:14:59.442914 12043 raft_consensus.cc:2866] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 20:14:59.407193 16689 log.cc:924] Time spent T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
W20260808 20:14:59.408504 16678 log.cc:924] Time spent T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
I20260808 20:14:59.411374 12247 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 10 FOLLOWER]: Advancing to term 11
W20260808 20:14:59.445477 16691 log.cc:924] Time spent T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.094s user 0.001s sys 0.000s
I20260808 20:14:59.420905 16692 raft_consensus.cc:2726] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 20:14:59.448758 12241 raft_consensus.cc:1226] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260808 20:14:59.450667 12206 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
W20260808 20:14:59.455229 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 238625us
I20260808 20:14:59.456282 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:14:59.465416 12247 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 11.
I20260808 20:14:59.468314 11994 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:14:59.451148 16670 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:14:59.453011 16438 consensus_meta.cc:287] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.677s user 0.007s sys 0.004s
I20260808 20:14:59.467154 16605 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:14:59.486315 12048 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 8.
I20260808 20:14:59.467844 16586 raft_consensus.cc:2781] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader election won for term 8
I20260808 20:14:59.487808 12036 raft_consensus.cc:1261] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 10 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260808 20:14:59.487767 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 164382us
I20260808 20:14:59.489209 12102 leader_election.cc:302] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
I20260808 20:14:59.477869 16670 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:14:59.477761 16508 raft_consensus.cc:2781] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 20:14:59.504501 12137 raft_consensus.cc:2454] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 8.
I20260808 20:14:59.469254 16601 consensus_queue.cc:1046] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260808 20:14:59.523316 12047 raft_consensus.cc:1261] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260808 20:14:59.516662 16586 raft_consensus.cc:686] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [term 8 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:14:59.518882 16508 raft_consensus.cc:686] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [term 11 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:59.501138 16605 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:59.547997 12140 raft_consensus.cc:1261] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260808 20:14:59.556272 16586 consensus_queue.cc:237] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:59.565941 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 45841.8us
W20260808 20:14:59.585063 16715 log.cc:924] Time spent T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20260808 20:14:59.586316 16605 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:59.585160 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:59.591089 13075 raft_consensus.cc:2454] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:14:59.535291 16574 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:14:59.597250 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 108126us
W20260808 20:14:59.610838 16719 log.cc:924] Time spent T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.074s user 0.001s sys 0.000s
I20260808 20:14:59.612843 16597 raft_consensus.cc:3032] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 20:14:59.613281 16597 raft_consensus.cc:729] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:14:59.594100 16438 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:59.616873 12245 raft_consensus.cc:1226] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 5. Current term is 6. Ops: []
I20260808 20:14:59.627789 16681 raft_consensus.cc:2781] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 20:14:59.589304 16675 consensus_queue.cc:1035] T efc388dadfe84be190c9ddd0b3725244 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260808 20:14:59.643193 16263 consensus_queue.cc:1046] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 20:14:59.662282 13015 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
W20260808 20:14:59.645483 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 52530.4us
I20260808 20:14:59.603873 16661 raft_consensus.cc:482] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:14:59.614379 16508 consensus_queue.cc:237] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:59.634702 16438 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:14:59.656402 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 181026us
W20260808 20:14:59.657378 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 59521.4us
I20260808 20:14:59.662660 12036 raft_consensus.cc:1226] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7. Ops: []
W20260808 20:14:59.656337 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 94408.3us
I20260808 20:14:59.661303 12040 raft_consensus.cc:1261] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260808 20:14:59.620277 16670 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:14:59.693534 13080 raft_consensus.cc:1261] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 20:14:59.694253 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:59.695928 12207 leader_election.cc:302] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:14:59.705489 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:14:59.676733 15908 consensus_queue.cc:1046] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260808 20:14:59.658056 16681 raft_consensus.cc:686] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:14:59.703531 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:59.723187 12048 raft_consensus.cc:1261] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260808 20:14:59.726217 12245 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 8.
I20260808 20:14:59.728381 16681 consensus_queue.cc:237] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:59.614081 16597 consensus_queue.cc:260] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:59.730075 16597 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:14:59.732230 16669 raft_consensus.cc:2781] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 20:14:59.729056 16712 raft_consensus.cc:2781] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:14:59.661094 16605 consensus_queue.cc:1035] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.066s
I20260808 20:14:59.744271 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:14:59.745347 16703 consensus_queue.cc:1046] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260808 20:14:59.685354 16661 raft_consensus.cc:504] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:59.746874 16580 raft_consensus.cc:3032] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Stepping down as leader of term 6
I20260808 20:14:59.747291 16580 raft_consensus.cc:729] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
W20260808 20:14:59.752411 16725 log.cc:924] Time spent T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.131s user 0.000s sys 0.001s
I20260808 20:14:59.733096 16649 consensus_queue.cc:1035] T 2a41c716cfc94652a2fc6957a1bd2a78 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.022s
I20260808 20:14:59.739145 16712 raft_consensus.cc:482] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:14:59.759490 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:14:59.731721 16400 consensus_queue.cc:1046] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260808 20:14:59.747385 16438 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:14:59.790565 13072 raft_consensus.cc:1203] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[4.5-5.6] Dedup: 4.5->[5.6-5.6]
I20260808 20:14:59.791131 13072 raft_consensus.cc:1226] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6. Ops: [4.5-5.6]
I20260808 20:14:59.764204 16712 raft_consensus.cc:3037] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
W20260808 20:14:59.812362 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 123381us
W20260808 20:14:59.816711 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 82779.5us
I20260808 20:14:59.787031 16574 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:14:59.957772 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 269463us
I20260808 20:14:59.954051 13016 leader_election.cc:302] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:15:00.006914 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 52953.7us
W20260808 20:15:00.007692 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 52445.1us
I20260808 20:14:59.812268 15674 raft_consensus.cc:482] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:15:00.015334 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 222685us
W20260808 20:15:00.020684 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 310368us
I20260808 20:14:59.996209 16609 raft_consensus.cc:482] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:15:00.012750 16644 raft_consensus.cc:3037] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:14:59.783975 16580 consensus_queue.cc:260] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:14:59.866818 16400 consensus_queue.cc:1035] T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.156s
I20260808 20:15:00.009055 16669 raft_consensus.cc:686] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:15:00.038371 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:00.000450 16588 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:00.017611 15592 raft_consensus.cc:482] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:00.006611 16501 consensus_queue.cc:1035] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260808 20:15:00.044759 16670 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:00.036321 16609 raft_consensus.cc:504] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:00.071715 16592 consensus_queue.cc:1035] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 20:15:00.025728 16649 consensus_queue.cc:1046] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260808 20:15:00.027446 15674 raft_consensus.cc:504] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:00.069941 16752 raft_consensus.cc:3032] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:15:00.126130 16508 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:15:00.127305 16728 log.cc:924] Time spent T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.113s user 0.000s sys 0.000s
W20260808 20:15:00.090932 16668 log.cc:924] Time spent T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
W20260808 20:15:00.152493 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 126975us
I20260808 20:15:00.087386 16388 raft_consensus.cc:3032] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 20:15:00.118785 16580 raft_consensus.cc:3037] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:00.169623 12039 raft_consensus.cc:1261] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260808 20:15:00.136062 16597 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:00.085191 15592 raft_consensus.cc:504] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:15:00.181886 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 143193us
I20260808 20:15:00.128630 16752 raft_consensus.cc:729] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:15:00.160118 16388 raft_consensus.cc:729] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
W20260808 20:15:00.195917 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 186660us
W20260808 20:15:00.193686 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 175421us
I20260808 20:15:00.075336 16588 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:00.159011 16661 leader_election.cc:288] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:15:00.201674 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 244616us
I20260808 20:15:00.161438 16400 consensus_queue.cc:1035] T c0f516c855714940a41eac002203e145 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 8, Time since last communication: 0.025s
W20260808 20:15:00.201774 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 167457us
I20260808 20:15:00.163025 16759 raft_consensus.cc:2726] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:15:00.229365 12101 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:15:00.155061 16508 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:00.196787 16735 consensus_queue.cc:1035] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260808 20:15:00.209021 13016 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:15:00.240815 12102 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:00.195426 16477 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:00.195326 16703 raft_consensus.cc:482] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
W20260808 20:15:00.254123 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 57092us
I20260808 20:15:00.218533 15592 leader_election.cc:288] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:00.196559 16597 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:00.165642 16669 consensus_queue.cc:237] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:00.264288 16542 consensus_queue.cc:1035] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260808 20:15:00.293841 16511 consensus_queue.cc:1035] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260808 20:15:00.294598 16750 log.cc:924] Time spent T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.320s user 0.000s sys 0.001s
I20260808 20:15:00.294756 16542 raft_consensus.cc:2726] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 20:15:00.295342 16542 consensus_queue.cc:1035] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
W20260808 20:15:00.306185 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 97274.5us
W20260808 20:15:00.322710 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 299274us
I20260808 20:15:00.295436 16511 consensus_queue.cc:1035] T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20260808 20:15:00.258414 16681 consensus_queue.cc:1035] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20260808 20:15:00.251507 16477 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:15:00.333245 12000 leader_election.cc:334] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.626s (SENT)
I20260808 20:15:00.279206 16574 raft_consensus.cc:482] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:00.328467 12243 raft_consensus.cc:1226] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 8 LEADER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 7. Current term is 8. Ops: []
W20260808 20:15:00.334134 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 104890us
I20260808 20:15:00.335857 12105 leader_election.cc:302] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:00.257160 16703 raft_consensus.cc:504] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:15:00.328491 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 75136.8us
I20260808 20:15:00.345175 12000 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:15:00.383273 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:00.384299 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:00.384819 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:00.385270 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:00.340344 16605 raft_consensus.cc:2781] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 20:15:00.391844 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:15:00.346415 16741 log.cc:924] Time spent T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.256s user 0.000s sys 0.001s
I20260808 20:15:00.339526 16609 leader_election.cc:288] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:00.420606 12146 raft_consensus.cc:2379] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 7: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:15:00.419203 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:00.425840 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:00.427652 12142 raft_consensus.cc:3037] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 10 FOLLOWER]: Advancing to term 11
I20260808 20:15:00.436529 16670 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:00.437077 16670 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:00.439106 16670 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:00.440026 16670 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:00.440622 16670 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:00.345578 15674 leader_election.cc:288] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:00.507259 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:00.508122 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:15:00.509521 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 176304us
W20260808 20:15:00.547065 12000 leader_election.cc:334] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.694s (SENT)
W20260808 20:15:00.548012 12000 leader_election.cc:334] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.722s (SENT)
I20260808 20:15:00.548388 12000 leader_election.cc:302] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:15:00.507259 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:00.543174 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:00.507807 12137 raft_consensus.cc:2379] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 7: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:15:00.543946 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:00.508648 15908 consensus_queue.cc:1046] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260808 20:15:00.559453 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:00.549149 12140 raft_consensus.cc:2362] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7. Re-sending same reply.
I20260808 20:15:00.426767 16605 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:15:00.262035 16388 consensus_queue.cc:260] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:00.419397 16599 raft_consensus.cc:2726] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 20:15:00.569501 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:00.568462 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:00.548964 16609 consensus_queue.cc:1035] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260808 20:15:00.571717 16749 raft_consensus.cc:2726] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 20:15:00.373638 16574 raft_consensus.cc:504] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:00.324401 16752 consensus_queue.cc:260] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:00.572587 16661 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:00.736307 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:00.573365 16605 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:00.743505 16814 raft_consensus.cc:2726] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:15:00.754387 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:00.755103 16794 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:00.755941 16767 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:00.757512 16677 raft_consensus.cc:482] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:00.759529 16670 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:00.761219 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:00.761639 12142 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 11.
I20260808 20:15:00.742007 16388 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:15:00.685294 16588 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:15:00.769728 16783 log.cc:924] Time spent T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.197s user 0.000s sys 0.000s
W20260808 20:15:00.770912 16770 log.cc:924] Time spent T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.209s user 0.000s sys 0.000s
W20260808 20:15:00.777159 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 460227us
I20260808 20:15:00.789331 12147 raft_consensus.cc:2362] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 8. Re-sending same reply.
I20260808 20:15:00.790196 12150 raft_consensus.cc:1226] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260808 20:15:00.684710 16599 raft_consensus.cc:2726] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:15:00.796977 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:00.799954 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:00.802322 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:00.803005 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:00.806882 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:00.684403 16508 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:00.829680 12046 raft_consensus.cc:1261] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 20:15:00.830502 12045 raft_consensus.cc:1261] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 20:15:00.830646 12144 raft_consensus.cc:2379] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 5: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:15:00.822645 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:00.811713 16677 raft_consensus.cc:504] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:00.807474 16597 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:00.831225 16746 raft_consensus.cc:3032] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:15:00.829988 16814 raft_consensus.cc:482] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:00.863881 12152 raft_consensus.cc:2379] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 8: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:15:00.864625 12150 raft_consensus.cc:1261] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260808 20:15:00.845458 16752 raft_consensus.cc:3037] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:15:00.858390 12036 raft_consensus.cc:2379] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 9: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:15:00.844249 16670 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
W20260808 20:15:00.866595 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 438752us
I20260808 20:15:00.794471 16661 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:00.853979 16786 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
W20260808 20:15:00.863979 16754 log.cc:924] Time spent T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.553s user 0.001s sys 0.001s
I20260808 20:15:00.882408 16746 raft_consensus.cc:729] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:15:00.877095 16508 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:15:00.820111 16767 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:00.856331 16775 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:00.892994 16477 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:00.893242 16786 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:00.914283 16599 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:00.810779 16692 raft_consensus.cc:482] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:00.828517 16794 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:15:01.013468 12205 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.757s user 0.000s sys 0.017s
W20260808 20:15:01.017230 12205 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.763s user 0.000s sys 0.018s
I20260808 20:15:00.891997 16670 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:00.935527 16703 leader_election.cc:288] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:15:01.009807 16754 log.cc:924] Time spent T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.096s user 0.004s sys 0.004s
W20260808 20:15:01.012472 16712 consensus_meta.cc:287] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 1.177s user 0.011s sys 0.007s
W20260808 20:15:01.062574 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 218820us
I20260808 20:15:00.886353 16814 raft_consensus.cc:504] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:15:01.057104 12247 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.727s user 0.027s sys 0.028s
W20260808 20:15:01.075685 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 527658us
I20260808 20:15:01.076221 12243 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "78b9263889cd4f1582578a3aa4f644d9"
from {username='slave'} at 127.0.0.1:40912
W20260808 20:15:01.082111 12206 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.855s user 0.010s sys 0.012s
W20260808 20:15:01.113575 12206 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.887s user 0.010s sys 0.012s
W20260808 20:15:01.113618 12000 leader_election.cc:334] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.728s (SENT)
I20260808 20:15:00.917440 16508 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:01.114142 12000 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:15:01.115718 12247 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.786s user 0.028s sys 0.028s
I20260808 20:15:01.113603 12243 raft_consensus.cc:595] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Received request to transfer leadership to TS 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:15:01.150168 12243 raft_consensus.cc:603] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 20:15:01.152700 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:01.059490 16574 leader_election.cc:288] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:01.153355 12045 raft_consensus.cc:2454] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 7.
I20260808 20:15:01.085287 16785 raft_consensus.cc:482] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:01.021510 16599 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:01.084820 16703 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
W20260808 20:15:01.233274 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 119693us
W20260808 20:15:01.234859 12000 leader_election.cc:334] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.549s (SENT)
I20260808 20:15:01.235258 12000 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:15:01.245021 12000 leader_election.cc:334] T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.792s (SENT)
I20260808 20:15:00.977408 16775 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:15:01.221757 16754 log.cc:924] Time spent T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
I20260808 20:15:01.214190 16005 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:01.269237 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:15:01.280421 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 378769us
I20260808 20:15:01.221186 16798 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
W20260808 20:15:01.357272 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 112286us
I20260808 20:15:01.095603 16692 raft_consensus.cc:504] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:01.239069 16597 raft_consensus.cc:2726] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 20:15:01.099190 16712 raft_consensus.cc:504] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:01.379874 16712 leader_election.cc:288] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:01.380517 16670 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:01.381122 16670 raft_consensus.cc:2726] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 20:15:01.381376 16809 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:01.382190 16700 raft_consensus.cc:482] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:01.382813 16700 raft_consensus.cc:504] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:15:01.384675 16753 log.cc:924] Time spent T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.590s user 0.000s sys 0.001s
I20260808 20:15:01.385887 12048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1585) took 1643 ms. Trace:
I20260808 20:15:01.386714 16799 raft_consensus.cc:482] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:01.387233 16799 raft_consensus.cc:504] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:01.388677 16835 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:01.384785 16700 leader_election.cc:288] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:01.389176 16799 leader_election.cc:288] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:01.389582 16700 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:01.389750 16799 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:01.390050 16700 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:01.390726 16830 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:01.391254 16830 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:01.392654 16700 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:01.393270 16700 raft_consensus.cc:482] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:01.393705 16700 raft_consensus.cc:504] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:01.394177 16794 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:01.395699 16508 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:01.380538 16438 raft_consensus.cc:482] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:15:01.396211 16508 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:01.396669 16508 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:01.397948 16814 leader_election.cc:288] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:01.398514 16508 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:01.399019 16508 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:01.399391 16599 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:01.399915 16599 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:01.401458 16767 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:01.388361 16592 consensus_queue.cc:1035] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:15:01.386219 12048 rpcz_store.cc:276] 0808 20:14:59.742286 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:59.742430 (+ 144us) service_pool.cc:227] Handling call
0808 20:14:59.743436 (+ 1006us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:59.744061 (+ 625us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:59.744090 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:59.744512 (+ 422us) log.cc:842] Serialized 26 byte log entry
0808 20:15:00.006754 (+262242us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:15:00.006799 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:00.006863 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:15:01.385683 (+1378820us) raft_consensus.cc:1671] finished
0808 20:15:01.385703 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:01.385811 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1286,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":75,"thread_start_us":261911,"threads_started":1,"wal-append.queue_time_us":516776}
I20260808 20:15:01.381886 16809 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:01.400328 16599 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:01.380434 16712 consensus_queue.cc:1035] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
W20260808 20:15:01.375692 12138 consensus_meta.cc:287] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.815s user 0.009s sys 0.000s
I20260808 20:15:01.423700 12138 raft_consensus.cc:2454] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
I20260808 20:15:01.435678 12045 raft_consensus.cc:2396] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 8 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260808 20:15:01.370491 16797 log.cc:924] Time spent T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.352s user 0.000s sys 0.000s
I20260808 20:15:01.456315 12138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1818) took 1049 ms. Trace:
I20260808 20:15:01.389298 16777 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:15:01.401494 16764 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.757s user 0.002s sys 0.014s
W20260808 20:15:01.460355 16793 log.cc:924] Time spent T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.059s user 0.000s sys 0.001s
W20260808 20:15:01.431181 16754 log.cc:924] Time spent T aa97e9c654da4bcda377ef63ec1c694b P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260808 20:15:01.243505 16703 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:01.403250 16677 leader_election.cc:288] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:01.390245 16799 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:01.396399 16438 raft_consensus.cc:504] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:01.370980 16661 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:01.528018 16830 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:01.078610 16746 consensus_queue.cc:260] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:15:01.540388 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 313153us
I20260808 20:15:01.316807 16005 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:15:01.508623 16764 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.261s user 0.002s sys 0.015s
I20260808 20:15:01.399407 16508 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:01.534238 16597 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:01.456842 12138 rpcz_store.cc:276] 0808 20:15:00.384455 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:00.384572 (+ 117us) service_pool.cc:227] Handling call
0808 20:15:01.433656 (+1049084us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:01.299775 16785 raft_consensus.cc:504] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:01.538163 16661 raft_consensus.cc:482] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:01.535420 12000 leader_election.cc:302] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:01.541388 12102 leader_election.cc:302] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:15:01.537763 16733 log.cc:924] Time spent T a83f6daa15a54982865040c0110bdda8 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 1.772s user 0.006s sys 0.024s
W20260808 20:15:01.548707 12142 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.656s user 0.000s sys 0.015s
I20260808 20:15:01.389209 16835 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:01.537647 16692 leader_election.cc:288] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:01.535562 16670 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:01.371179 16798 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:01.535851 16710 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:15:01.536547 16775 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:01.538889 16786 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:15:01.592976 12046 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.701s user 0.032s sys 0.058s
W20260808 20:15:01.614718 12046 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.725s user 0.033s sys 0.058s
I20260808 20:15:01.399410 16700 leader_election.cc:288] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:15:01.586481 12142 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.694s user 0.001s sys 0.015s
W20260808 20:15:01.605600 13016 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.002s sys 0.036s
W20260808 20:15:01.627197 13016 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.528s user 0.002s sys 0.037s
W20260808 20:15:01.606810 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 71468.6us
I20260808 20:15:01.565714 16677 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:01.629261 12000 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:01.565879 16824 raft_consensus.cc:482] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:01.570331 16764 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:01.557125 16521 consensus_queue.cc:1035] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.052s
W20260808 20:15:01.597366 16669 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.008s sys 0.024s
I20260808 20:15:01.652974 12205 leader_election.cc:302] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:15:01.604861 16738 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:01.648391 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:01.660392 13080 raft_consensus.cc:2454] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
W20260808 20:15:01.669893 12206 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.004s sys 0.010s
W20260808 20:15:01.670993 12206 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.557s user 0.004s sys 0.010s
I20260808 20:15:01.637236 16692 raft_consensus.cc:2726] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 20:15:01.631449 16501 raft_consensus.cc:2726] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:15:01.632550 16786 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:01.565109 16597 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:01.585064 16661 raft_consensus.cc:504] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:01.665953 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:15:01.654052 16669 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.576s user 0.009s sys 0.024s
W20260808 20:15:01.676688 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 47514.7us
I20260808 20:15:01.726913 12000 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:15:01.679826 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:01.682657 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:01.685716 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:01.728099 12250 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:01.513391 16777 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:01.613318 16710 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:01.615203 16670 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:01.666007 16775 raft_consensus.cc:2781] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260808 20:15:01.723591 16731 log.cc:924] Time spent T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 1.977s user 0.005s sys 0.016s
W20260808 20:15:01.685770 12038 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.010s sys 0.010s
I20260808 20:15:01.716681 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:01.728816 12249 raft_consensus.cc:1261] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 11 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260808 20:15:01.721825 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:01.720619 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:15:01.681097 16674 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.056s user 0.014s sys 0.019s
I20260808 20:15:01.772249 13079 raft_consensus.cc:2379] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 8: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:15:01.641738 16677 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:01.682807 16830 raft_consensus.cc:482] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:01.739439 13078 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8.
I20260808 20:15:01.673488 16864 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:01.730913 16817 raft_consensus.cc:2726] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:15:01.683663 16814 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:15:01.767668 16819 log.cc:924] Time spent T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.329s user 0.000s sys 0.001s
I20260808 20:15:01.653381 16867 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:15:01.767365 12038 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.706s user 0.010s sys 0.010s
I20260808 20:15:01.717188 16868 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:01.717525 16599 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:15:01.740741 16752 consensus_meta.cc:287] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.853s user 0.008s sys 0.002s
I20260808 20:15:01.726251 16809 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:01.722616 16835 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:15:01.752595 12041 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.864s user 0.003s sys 0.020s
W20260808 20:15:01.895147 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 233285us
W20260808 20:15:01.772408 16674 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.147s user 0.015s sys 0.021s
I20260808 20:15:01.725382 16859 consensus_queue.cc:1046] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.086s
W20260808 20:15:01.915339 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 130869us
I20260808 20:15:01.915702 16817 raft_consensus.cc:482] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:01.919440 12142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 459) took 1115 ms. Trace:
I20260808 20:15:01.767436 16775 raft_consensus.cc:482] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:15:01.918077 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 191231us
I20260808 20:15:01.969054 12142 rpcz_store.cc:276] 0808 20:15:00.803554 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:00.822090 (+ 18536us) service_pool.cc:227] Handling call
0808 20:15:00.822416 (+ 326us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:15:00.838552 (+ 16136us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 20:15:00.866206 (+ 27654us) log.cc:842] Serialized 14 byte log entry
0808 20:15:01.393522 (+527316us) spinlock_profiling.cc:243] Waited 4.39 ms on lock 0x7b18001f0ba0. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96e2518b1 00007fa96e25e661 00007fa9653b2764 00007fa9653b5b11 00007fa9653b5ac9 00007fa9653b5a59 00007fa9653b5a21 00007fa9653b4d1d 00007fa96e616204 00007fa96e616039 00007fa965394dc6
0808 20:15:01.626827 (+233305us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:15:01.628090 (+ 1263us) log.cc:842] Serialized 26 byte log entry
0808 20:15:01.628211 (+ 121us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:15:01.628412 (+ 201us) log.cc:842] Serialized 14 byte log entry
0808 20:15:01.628560 (+ 148us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:01.628629 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:15:01.919215 (+290586us) raft_consensus.cc:1671] finished
0808 20:15:01.919236 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:01.919357 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":36615,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":5184,"raft.queue_time_us":246744,"raft.run_cpu_time_us":340,"raft.run_wall_time_us":34510,"spinlock_wait_cycles":10096000,"thread_start_us":733973,"threads_started":1,"wal-append.queue_time_us":1038424}
I20260808 20:15:01.978246 13016 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:15:01.751472 16703 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:01.725782 16669 consensus_queue.cc:1035] T c4dc15f09b4a4470ac96054b70602187 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.061s
I20260808 20:15:01.922053 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 2111) took 1122 ms. Trace:
W20260808 20:15:01.929870 16388 consensus_meta.cc:287] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 1.091s user 0.010s sys 0.000s
I20260808 20:15:01.673905 16738 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:01.713030 16786 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:15:01.927987 16803 log.cc:924] Time spent T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.196s user 0.000s sys 0.001s
W20260808 20:15:01.928745 16851 log.cc:924] Time spent T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.292s user 0.000s sys 0.000s
I20260808 20:15:01.656998 16764 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:15:02.001498 12251 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8.
W20260808 20:15:01.935716 12207 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.691s user 0.022s sys 0.044s
W20260808 20:15:02.002409 12207 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.758s user 0.023s sys 0.044s
I20260808 20:15:01.920274 16839 raft_consensus.cc:482] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:01.915702 16855 raft_consensus.cc:482] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:01.915939 16856 raft_consensus.cc:482] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:01.938910 13073 raft_consensus.cc:2454] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:15:01.949779 12250 raft_consensus.cc:2454] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 9.
I20260808 20:15:01.984009 16775 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:15:01.947551 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:15:01.938952 12041 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.051s user 0.003s sys 0.020s
I20260808 20:15:01.972379 16599 consensus_queue.cc:1035] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 20:15:01.973528 16712 raft_consensus.cc:482] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:01.973469 16873 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:01.916765 16677 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:01.977337 12000 leader_election.cc:302] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:15:01.984531 12001 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:15:01.986259 16798 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:01.986692 16818 raft_consensus.cc:482] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:01.987143 16863 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:01.990291 12047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 637) took 2428 ms. Trace:
I20260808 20:15:01.990439 13074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53296 (request call id 795) took 2310 ms. Trace:
I20260808 20:15:01.986279 16777 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:01.991315 16592 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:01.993427 16859 raft_consensus.cc:3032] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 20:15:01.922039 16799 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:01.660064 16824 raft_consensus.cc:504] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:01.997443 12138 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:49548
I20260808 20:15:01.983078 12038 rpcz_store.cc:276] 0808 20:15:00.799868 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:00.799975 (+ 107us) service_pool.cc:227] Handling call
0808 20:15:00.828786 (+ 28811us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:00.829016 (+ 230us) raft_consensus.cc:1520] Early marking committed up to index 5
0808 20:15:01.061191 (+232175us) log.cc:842] Serialized 14 byte log entry
0808 20:15:01.921742 (+860551us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:01.921794 (+ 52us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:15:01.921835 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:01.921900 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:01.922006 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":666140,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":3585,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":2192,"thread_start_us":1064284,"threads_started":2}
I20260808 20:15:01.930264 16438 leader_election.cc:288] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:01.938411 16785 leader_election.cc:288] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:02.175664 12138 raft_consensus.cc:595] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:15:01.984337 16817 raft_consensus.cc:504] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:02.217901 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:02.219237 16817 leader_election.cc:288] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:01.906889 16830 raft_consensus.cc:504] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:02.221019 16830 leader_election.cc:288] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:01.983793 16508 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:02.221174 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.222546 16661 leader_election.cc:288] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:02.101465 16712 raft_consensus.cc:504] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:02.223732 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.224613 16712 leader_election.cc:288] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:02.062933 12048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 2120) took 1011 ms. Trace:
W20260808 20:15:02.082382 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 105142us
I20260808 20:15:02.225507 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.225809 16802 raft_consensus.cc:2726] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:15:02.223304 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 11 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.227718 16824 leader_election.cc:288] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:02.228065 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:02.228245 16853 raft_consensus.cc:482] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:02.228960 16853 raft_consensus.cc:504] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:01.997601 16703 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:02.229954 12000 leader_election.cc:302] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:15:02.230098 16703 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:02.230826 16796 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:15:02.231590 16796 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:02.232287 16703 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:02.139222 16692 raft_consensus.cc:2726] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:15:02.175738 16597 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:02.161545 16777 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:02.233649 16796 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:02.233844 16710 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:02.234077 16777 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:15:02.099397 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 114972us
W20260808 20:15:02.099448 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 94531.3us
I20260808 20:15:02.225464 12048 rpcz_store.cc:276] 0808 20:15:01.051910 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:01.436775 (+384865us) service_pool.cc:227] Handling call
0808 20:15:01.481139 (+ 44364us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:15:01.481504 (+ 365us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:01.481529 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:15:01.481954 (+ 425us) log.cc:842] Serialized 26 byte log entry
0808 20:15:01.726954 (+245000us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:15:01.769348 (+ 42394us) log.cc:842] Serialized 14 byte log entry
0808 20:15:01.769555 (+ 207us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:01.769625 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:15:02.062694 (+293069us) raft_consensus.cc:1671] finished
0808 20:15:02.062715 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:02.062859 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":40135,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":25826,"raft.queue_time_us":16960,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":18561,"spinlock_wait_cycles":14659328,"thread_start_us":196627,"threads_started":1,"wal-append.queue_time_us":494310}
I20260808 20:15:02.099308 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:02.102006 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:01.974982 16759 consensus_queue.cc:1035] T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.108s
I20260808 20:15:02.259927 12138 raft_consensus.cc:2379] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 4: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:15:02.081450 16388 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:02.083638 16861 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:15:01.930382 16867 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:01.916978 16814 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:02.229434 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.140944 12047 rpcz_store.cc:276] 0808 20:14:59.561500 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:59.561633 (+ 133us) service_pool.cc:227] Handling call
0808 20:14:59.577493 (+ 15860us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:14:59.577845 (+ 352us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:14:59.577873 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:14:59.578255 (+ 382us) log.cc:842] Serialized 26 byte log entry
0808 20:14:59.806563 (+228308us) raft_consensus.cc:1631] Marking committed up to 8
0808 20:15:00.000504 (+193941us) log.cc:842] Serialized 14 byte log entry
0808 20:15:00.000992 (+ 488us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:00.001112 (+ 120us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:15:01.380195 (+1379083us) spinlock_profiling.cc:243] Waited 8.88 ms on lock 0x7b18002f6020. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96e2518b1 00007fa96e25e661 00007fa9653b2638 00007fa9653b5b11 00007fa9653b5ac9 00007fa9653b5a59 00007fa9653b5a21 00007fa9653b4d1d 00007fa96e616204 00007fa96e616039 00007fa965394dc6
0808 20:15:01.988281 (+608086us) spinlock_profiling.cc:243] Waited 2.8 ms on lock 0x7b18002f6020. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96e2518b1 00007fa96e25e661 00007fa9653b2638 00007fa9653b5b11 00007fa9653b5ac9 00007fa9653b5a59 00007fa9653b5a21 00007fa9653b4d1d 00007fa96e616204 00007fa96e616039 00007fa965394dc6
0808 20:15:01.990081 (+ 1800us) raft_consensus.cc:1671] finished
0808 20:15:01.990103 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:01.990218 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":195762,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":231,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":7755,"spinlock_wait_cycles":26867712,"thread_start_us":207406,"threads_started":1,"wal-append.queue_time_us":430280}
I20260808 20:15:02.228860 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.224835 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.236512 12150 raft_consensus.cc:3037] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 6
I20260808 20:15:02.225325 16764 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:02.274176 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.273689 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:02.139442 13074 rpcz_store.cc:276] 0808 20:14:59.679565 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:59.692655 (+ 13090us) service_pool.cc:227] Handling call
0808 20:14:59.724697 (+ 32042us) raft_consensus.cc:1457] Updating replica for 3 ops
0808 20:15:00.033550 (+308853us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:00.033582 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0808 20:15:00.040010 (+ 6428us) log.cc:842] Serialized 78 byte log entry
0808 20:15:00.153435 (+113425us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:15:00.165479 (+ 12044us) log.cc:842] Serialized 14 byte log entry
0808 20:15:00.181525 (+ 16046us) log.cc:842] Serialized 14 byte log entry
0808 20:15:00.185091 (+ 3566us) log.cc:842] Serialized 14 byte log entry
0808 20:15:00.185361 (+ 270us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:00.185431 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:15:01.938050 (+1752619us) spinlock_profiling.cc:243] Waited 7.63 ms on lock 0x7b1800482b00. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96e2518b1 00007fa96e25e661 00007fa9653294ca 00007fa96ea0af99 00007fa96d181088 00007fa96d1786b1 00007fa96d178657 00007fa96d26c978 00007fa96d26c627 00007fa96d280b2c 00007fa96d280ad8
0808 20:15:01.990199 (+ 52149us) raft_consensus.cc:1671] finished
0808 20:15:01.990221 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:01.990362 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1715893,"prepare.queue_time_us":24306,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":93,"raft.queue_time_us":389326,"raft.run_cpu_time_us":17324,"raft.run_wall_time_us":3474288,"spinlock_wait_cycles":17553792,"thread_start_us":353642,"threads_started":2,"wal-append.queue_time_us":1883531}
W20260808 20:15:02.228240 16858 log.cc:924] Time spent T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.219s user 0.000s sys 0.000s
I20260808 20:15:01.964980 16868 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:15:01.925236 13018 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.015s sys 0.026s
I20260808 20:15:02.018905 16839 raft_consensus.cc:504] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:02.260365 12001 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:02.219237 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:02.271935 12146 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:15:02.267127 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.272832 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:02.228374 16899 raft_consensus.cc:2781] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:15:02.230162 16881 raft_consensus.cc:482] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:02.159459 16859 raft_consensus.cc:729] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
W20260808 20:15:02.363629 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 103335us
W20260808 20:15:02.355470 11994 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.24.3:38391 timed out after 2.130s (SENT) [suppressed 62 similar messages]
I20260808 20:15:02.355836 13078 raft_consensus.cc:2454] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
W20260808 20:15:02.276528 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 298332us
I20260808 20:15:02.274821 12140 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8.
I20260808 20:15:02.356699 12137 raft_consensus.cc:1261] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 11 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260808 20:15:02.282117 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:02.377239 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.377926 12146 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:15:02.277182 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.377250 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.378809 12144 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 9.
I20260808 20:15:02.361284 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:15:02.357543 13018 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.142s user 0.016s sys 0.037s
I20260808 20:15:02.362732 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:02.260476 16508 raft_consensus.cc:2726] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:15:02.359818 16005 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:02.355726 12038 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:15:02.384555 13016 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:15:02.381218 12000 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:15:02.233557 16738 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:02.271006 16661 raft_consensus.cc:482] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:02.288980 16796 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:02.386304 12048 raft_consensus.cc:2379] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 7: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:15:02.390465 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:02.390722 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:15:02.391170 16880 log.cc:924] Time spent T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.216s user 0.000s sys 0.000s
I20260808 20:15:02.063267 16855 raft_consensus.cc:504] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:02.355463 16597 raft_consensus.cc:482] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:02.391387 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:02.376576 16830 raft_consensus.cc:2726] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:15:02.376777 16817 raft_consensus.cc:482] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
W20260808 20:15:02.392304 16801 log.cc:924] Time spent T e5dc2ab0712f4054883178f4b4c50928 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.157s user 0.000s sys 0.000s
I20260808 20:15:02.393173 16786 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:02.393592 16777 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:02.377609 16438 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:02.394876 16868 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:02.369210 16710 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
W20260808 20:15:02.495065 16888 log.cc:924] Time spent T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.293s user 0.000s sys 0.000s
I20260808 20:15:02.494793 16710 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:02.063452 16856 raft_consensus.cc:504] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:02.397221 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.496486 12144 raft_consensus.cc:2454] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8.
W20260808 20:15:02.397749 16644 consensus_meta.cc:287] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 2.360s user 0.011s sys 0.000s
I20260808 20:15:02.164088 16818 raft_consensus.cc:504] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:15:02.471163 16879 log.cc:924] Time spent T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
I20260808 20:15:02.379799 16881 raft_consensus.cc:504] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:02.398645 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:15:02.399073 16605 consensus_meta.cc:287] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 1.569s user 0.007s sys 0.004s
I20260808 20:15:02.399317 16744 consensus_queue.cc:1035] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 20:15:02.399230 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:02.392218 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:02.390465 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.263387 16824 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:02.188560 16592 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:02.510546 12150 raft_consensus.cc:2454] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:15:02.417029 12000 leader_election.cc:302] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
W20260808 20:15:02.512858 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 95980.2us
I20260808 20:15:02.513052 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.264295 16670 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:02.418812 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.355876 16899 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:02.383996 12001 leader_election.cc:302] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:15:02.465178 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.100569 16873 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:02.467098 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:15:02.419507 16580 consensus_meta.cc:287] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 2.249s user 0.010s sys 0.002s
I20260808 20:15:02.267822 16388 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:02.479611 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:02.486096 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:02.419756 16799 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:02.494091 16839 leader_election.cc:288] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:02.465833 16508 raft_consensus.cc:2781] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260808 20:15:02.496060 16884 log.cc:924] Time spent T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.218s user 0.000s sys 0.000s
W20260808 20:15:02.497881 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 113414us
I20260808 20:15:02.398008 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.501555 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.541514 12144 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 8.
I20260808 20:15:02.502048 11994 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:02.171229 16863 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:02.267904 16861 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:02.553481 16861 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:02.502527 12047 raft_consensus.cc:2454] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:15:02.497836 16005 raft_consensus.cc:2726] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 20:15:02.553476 16863 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:02.554051 16830 raft_consensus.cc:2726] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 20:15:02.554414 16831 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:15:02.467814 16796 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:02.502976 16438 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:02.487025 16817 raft_consensus.cc:504] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:02.555104 16831 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:02.556376 16796 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:02.556704 16817 leader_election.cc:288] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:02.511142 16818 leader_election.cc:288] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:02.500075 16701 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:02.511266 16824 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:02.557555 16701 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:02.558465 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:02.558797 16824 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:02.513506 12000 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:15:02.559367 12047 raft_consensus.cc:2454] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:15:02.517876 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.560464 16831 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:02.537791 16508 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:15:02.459048 16661 raft_consensus.cc:504] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:02.561220 16508 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:02.561718 12247 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 625) took 2233 ms. Trace:
I20260808 20:15:02.519165 16670 raft_consensus.cc:2781] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:15:02.562039 12247 rpcz_store.cc:276] 0808 20:15:00.327899 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:00.328979 (+ 1080us) service_pool.cc:227] Handling call
0808 20:15:00.329249 (+ 270us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:00.329428 (+ 179us) raft_consensus.cc:1520] Early marking committed up to index 8
0808 20:15:00.329724 (+ 296us) log.cc:842] Serialized 14 byte log entry
0808 20:15:02.561279 (+2231555us) spinlock_profiling.cc:243] Waited 9.17 ms on lock 0x7b18003e58a0. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96e2518b1 00007fa96e25e661 00007fa967f6982c 00007fa9653949dc 00007fa96e1cba39 00007fa9653b18bb 00007fa9653afcbf 00007fa9653af155 00007fa9682d23d0 00007fa9682e4576 00007fa9682f50d7
0808 20:15:02.561369 (+ 90us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:02.561416 (+ 47us) raft_consensus.cc:1631] Marking committed up to 8
0808 20:15:02.561471 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:02.561546 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:02.561648 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20757,"prepare.queue_time_us":854,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":348408,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":19171,"spinlock_wait_cycles":21080832,"thread_start_us":1197288,"threads_started":2,"wal-append.queue_time_us":1330101}
I20260808 20:15:02.563654 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.564178 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:02.519904 16592 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:02.520413 16899 raft_consensus.cc:3037] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:02.520942 12254 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 9.
I20260808 20:15:02.559057 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:02.566819 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:02.567379 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.567530 12254 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:15:02.567966 12245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 619) took 2241 ms. Trace:
I20260808 20:15:02.562665 16670 raft_consensus.cc:482] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:15:02.523372 16907 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:02.522269 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 2114) took 1718 ms. Trace:
I20260808 20:15:02.569641 16907 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:02.569662 12041 rpcz_store.cc:276] 0808 20:15:00.804241 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:00.804365 (+ 124us) service_pool.cc:227] Handling call
0808 20:15:00.828928 (+ 24563us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:15:00.851246 (+ 22318us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:00.851275 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:15:00.865479 (+ 14204us) log.cc:842] Serialized 26 byte log entry
0808 20:15:02.023100 (+1157621us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:15:02.099851 (+ 76751us) log.cc:842] Serialized 14 byte log entry
0808 20:15:02.100070 (+ 219us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:02.100149 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:15:02.522077 (+421928us) raft_consensus.cc:1671] finished
0808 20:15:02.522097 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:02.522218 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8108,"prepare.queue_time_us":105794,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":3473,"raft.queue_time_us":145597,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":11853,"spinlock_wait_cycles":16735616,"thread_start_us":1134841,"threads_started":1,"wal-append.queue_time_us":1138206}
I20260808 20:15:02.571718 16907 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:02.560649 12146 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:15:02.508232 16605 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:15:02.576558 16813 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:02.572568 16334 raft_consensus.cc:3032] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:15:02.574306 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:02.522538 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.525449 16868 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:02.525748 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:02.525950 16814 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:02.497243 16859 consensus_queue.cc:260] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:02.527930 13075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53296 (request call id 796) took 1709 ms. Trace:
I20260808 20:15:02.528146 16853 leader_election.cc:288] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:02.516062 16644 raft_consensus.cc:2781] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:15:02.568342 12245 rpcz_store.cc:276] 0808 20:15:00.326164 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:00.326298 (+ 134us) service_pool.cc:227] Handling call
0808 20:15:00.326531 (+ 233us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:00.544737 (+218206us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:00.544768 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:00.544808 (+ 40us) raft_consensus.cc:1631] Marking committed up to 12
0808 20:15:00.544842 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:02.567761 (+2022919us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x7b540007e754. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:15:02.567805 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:02.567916 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":501282048}
I20260808 20:15:02.579691 16868 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:15:02.529455 16758 log.cc:924] Time spent T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 2.273s user 0.030s sys 0.021s
I20260808 20:15:02.521376 16875 raft_consensus.cc:482] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
W20260808 20:15:02.530823 16872 log.cc:924] Time spent T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260808 20:15:02.531412 12250 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
I20260808 20:15:02.521847 12048 raft_consensus.cc:2454] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:15:02.536628 12045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 2121) took 1460 ms. Trace:
I20260808 20:15:02.535946 12205 leader_election.cc:302] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:15:02.478559 16597 raft_consensus.cc:504] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:02.539002 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 11 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:02.543704 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:15:02.544682 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 106940us
I20260808 20:15:02.550626 16802 raft_consensus.cc:482] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:02.580116 16644 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:15:02.578158 16334 raft_consensus.cc:729] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:15:02.554967 12140 raft_consensus.cc:2379] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 8: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:15:02.557034 12152 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8.
I20260808 20:15:02.580358 13075 rpcz_store.cc:276] 0808 20:15:00.818034 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:00.818166 (+ 132us) service_pool.cc:227] Handling call
0808 20:15:00.851369 (+ 33203us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 20:15:01.113748 (+262379us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:01.113777 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 20:15:01.152409 (+ 38632us) log.cc:842] Serialized 52 byte log entry
0808 20:15:01.608657 (+456248us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:15:01.946627 (+337970us) log.cc:842] Serialized 14 byte log entry
0808 20:15:01.948492 (+ 1865us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:01.948573 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:15:02.527691 (+579118us) raft_consensus.cc:1671] finished
0808 20:15:02.527709 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:02.527852 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":963656,"prepare.queue_time_us":64704,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":226393,"raft.run_cpu_time_us":422,"raft.run_wall_time_us":951237,"thread_start_us":872206,"threads_started":3,"wal-append.queue_time_us":765674}
I20260808 20:15:02.564376 12138 raft_consensus.cc:2379] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 7: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:15:02.565440 16712 raft_consensus.cc:482] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:02.569190 16670 raft_consensus.cc:3037] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:02.577950 16813 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:02.577699 12247 raft_consensus.cc:2454] T b7e4c0cacf074dee848eed1ab4d27ebf P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:15:02.582459 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:02.562148 16438 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:02.520653 16873 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
W20260808 20:15:02.582748 12207 connection.cc:232] RPC call timeout handler was delayed by -1.1788744472432882s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1724s for a response
I20260808 20:15:02.579041 12244 raft_consensus.cc:2454] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:15:02.518710 16881 leader_election.cc:288] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:02.571858 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:02.568235 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:02.632126 12038 raft_consensus.cc:2454] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
I20260808 20:15:02.630273 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.577382 12043 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1588) took 1558 ms. Trace:
I20260808 20:15:02.568395 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:02.579461 12150 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:15:02.639101 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.519613 12142 raft_consensus.cc:2454] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8.
I20260808 20:15:02.646833 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:02.647040 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:02.539487 16799 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:02.648852 12147 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 460) took 1799 ms. Trace:
I20260808 20:15:02.649031 16706 raft_consensus.cc:482] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:02.649614 16799 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:15:02.522346 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 138435us
I20260808 20:15:02.567976 12137 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:15:02.516211 16675 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:02.589627 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:02.591070 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:02.593689 11994 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:15:02.553900 16798 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:15:02.595850 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 60001.3us
I20260808 20:15:02.537740 16777 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:15:02.559540 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 46113.6us
I20260808 20:15:02.604043 12141 raft_consensus.cc:2379] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 9: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:15:02.603024 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:02.604624 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:02.603629 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:02.579907 16867 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:02.592094 12045 rpcz_store.cc:276] 0808 20:15:01.060846 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:01.228214 (+167368us) service_pool.cc:227] Handling call
0808 20:15:02.521011 (+1292797us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b5400485dd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:15:02.521110 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":313635712}
I20260808 20:15:02.572458 16599 raft_consensus.cc:2781] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 20:15:02.575901 16856 leader_election.cc:288] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:02.627424 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:02.574196 16388 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:02.647183 16855 leader_election.cc:288] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:02.638232 12254 raft_consensus.cc:2454] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:15:02.651255 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:15:02.509712 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 93416.9us
W20260808 20:15:02.695655 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 191563us
I20260808 20:15:02.653306 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
W20260808 20:15:02.745513 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 151908us
I20260808 20:15:02.684542 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:02.649240 12147 rpcz_store.cc:276] 0808 20:15:00.849618 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:00.855042 (+ 5424us) service_pool.cc:227] Handling call
0808 20:15:00.863716 (+ 8674us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 20:15:00.872415 (+ 8699us) raft_consensus.cc:1520] Early marking committed up to index 10
0808 20:15:01.227886 (+355471us) log.cc:842] Serialized 14 byte log entry
0808 20:15:02.071061 (+843175us) log.cc:842] Serialized 14 byte log entry
0808 20:15:02.200645 (+129584us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 20:15:02.228692 (+ 28047us) log.cc:842] Serialized 52 byte log entry
0808 20:15:02.228824 (+ 132us) raft_consensus.cc:1631] Marking committed up to 12
0808 20:15:02.229062 (+ 238us) log.cc:842] Serialized 14 byte log entry
0808 20:15:02.229331 (+ 269us) log.cc:842] Serialized 14 byte log entry
0808 20:15:02.229460 (+ 129us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:02.229530 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:15:02.648657 (+419127us) raft_consensus.cc:1671] finished
0808 20:15:02.648677 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:02.648800 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":181861,"prepare.queue_time_us":23519,"prepare.run_cpu_time_us":137,"prepare.run_wall_time_us":18477,"raft.queue_time_us":675764,"raft.run_cpu_time_us":501,"raft.run_wall_time_us":24751,"thread_start_us":1128729,"threads_started":3,"wal-append.queue_time_us":1121484}
I20260808 20:15:02.706301 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:02.571964 16899 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:02.825385 13077 raft_consensus.cc:1261] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260808 20:15:02.828054 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:02.826421 13015 leader_election.cc:302] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:15:02.833166 12249 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
W20260808 20:15:02.833554 16737 log.cc:924] Time spent T 00000000000000000000000000000000 P 9da0bf47b60c4880bc83222a7feaf045: Append to log took a long time: real 2.720s user 0.016s sys 0.019s
I20260808 20:15:02.833739 16853 raft_consensus.cc:2781] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:15:02.830232 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 2117) took 1907 ms. Trace:
I20260808 20:15:02.834201 16853 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:02.831607 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:02.825178 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:02.835078 16853 raft_consensus.cc:3037] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:15:02.835520 12045 raft_consensus.cc:2454] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 8.
I20260808 20:15:02.836400 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:02.833757 12249 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1939) took 1079 ms. Trace:
I20260808 20:15:02.837401 12045 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 8.
I20260808 20:15:02.824930 16908 consensus_queue.cc:1035] T 0285ac4646ab40c3b248ff959bbcd41e P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.022s
I20260808 20:15:02.838279 16813 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:02.838407 12102 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:15:02.839473 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:02.839908 15674 raft_consensus.cc:2781] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 20:15:02.840408 15674 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:15:02.841012 15674 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:02.834339 12040 rpcz_store.cc:276] 0808 20:15:00.923059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:00.924242 (+ 1183us) service_pool.cc:227] Handling call
0808 20:15:00.924565 (+ 323us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:15:00.961283 (+ 36718us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:00.961318 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:15:01.017747 (+ 56429us) log.cc:842] Serialized 26 byte log entry
0808 20:15:01.459715 (+441968us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:15:01.591575 (+131860us) log.cc:842] Serialized 14 byte log entry
0808 20:15:02.829585 (+1238010us) spinlock_profiling.cc:243] Waited 7.5 ms on lock 0x7b18004209a0. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96e2518b1 00007fa96e25e661 00007fa9653294ca 00007fa96ea0af99 00007fa967f77e10 00007fa967f6b663 00007fa967f6102a 00007fa967f6157b 00007fa9682e488a 00007fa9682f3f0f 00007fa9682f3e88
0808 20:15:02.829838 (+ 253us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:02.829911 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:15:02.829953 (+ 42us) raft_consensus.cc:1671] finished
0808 20:15:02.830048 (+ 95us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:02.830175 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":780447,"prepare.queue_time_us":133571,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":14739,"raft.queue_time_us":46208,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":12584,"spinlock_wait_cycles":17249664,"thread_start_us":441653,"threads_started":1,"wal-append.queue_time_us":563497}
I20260808 20:15:02.842271 16777 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:02.844147 16777 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:02.840054 12045 raft_consensus.cc:2454] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:15:02.844799 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:02.845640 12048 raft_consensus.cc:2454] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 6.
I20260808 20:15:02.843580 16334 consensus_queue.cc:260] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:02.634974 12043 rpcz_store.cc:276] 0808 20:15:01.019393 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:01.398718 (+379325us) service_pool.cc:227] Handling call
0808 20:15:01.399028 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:01.559359 (+160331us) raft_consensus.cc:1520] Early marking committed up to index 7
0808 20:15:01.559651 (+ 292us) log.cc:842] Serialized 14 byte log entry
0808 20:15:01.773168 (+213517us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:01.773213 (+ 45us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:15:01.773255 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:02.576051 (+802796us) spinlock_profiling.cc:243] Waited 27.5 ms on lock 0x7b1800392500. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96e2518b1 00007fa96e25e661 00007fa96537ff96 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9
0808 20:15:02.577156 (+ 1105us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b540038ccd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:15:02.577204 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:02.577325 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3207,"prepare.queue_time_us":135,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":72,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":33706,"spinlock_wait_cycles":431522816,"thread_start_us":213202,"threads_started":1,"wal-append.queue_time_us":521589}
I20260808 20:15:02.846706 12102 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:15:02.759645 12205 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:15:02.837318 12249 rpcz_store.cc:276] 0808 20:15:01.754002 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:01.779399 (+ 25397us) service_pool.cc:227] Handling call
0808 20:15:02.827908 (+1048509us) spinlock_profiling.cc:243] Waited 25.3 ms on lock 0x7b58001a1398. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:15:02.833636 (+ 5728us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":58164992}
I20260808 20:15:02.835413 13076 raft_consensus.cc:2396] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260808 20:15:02.746201 12000 leader_election.cc:302] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:15:02.861681 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 115522us
I20260808 20:15:02.845731 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:02.733659 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:02.885143 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:02.885815 12045 raft_consensus.cc:2454] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 8.
I20260808 20:15:02.886659 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:02.885435 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:02.872627 12047 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 7.
I20260808 20:15:02.889946 13014 leader_election.cc:302] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:15:02.898895 12000 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:02.604007 16644 raft_consensus.cc:3037] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:02.910729 13075 raft_consensus.cc:2396] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 8 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260808 20:15:02.912935 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:02.846112 12040 raft_consensus.cc:2454] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:15:02.912951 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:15:02.937654 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 47798us
W20260808 20:15:02.820618 16864 consensus_meta.cc:287] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.904s user 0.010s sys 0.000s
I20260808 20:15:02.782946 16438 raft_consensus.cc:2781] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 20:15:02.943477 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:02.650143 16910 raft_consensus.cc:2726] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:15:02.944072 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:02.596261 16875 raft_consensus.cc:504] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:02.834390 16868 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:02.631484 16712 raft_consensus.cc:504] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:02.626183 16802 raft_consensus.cc:504] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:02.685268 16675 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:15:04.398008 12000 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.615s user 0.005s sys 0.011s
I20260808 20:15:04.546900 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:15:04.507977 13018 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.721s user 0.016s sys 0.021s
I20260808 20:15:03.360383 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:02.944221 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:02.966100 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:02.960579 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:15:03.801103 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 1.04148e+06us
W20260808 20:15:04.551788 12152 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1847) took 1650 ms (client timeout 1791 ms). Trace:
W20260808 20:15:03.725627 12137 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.870s user 0.015s sys 0.004s
I20260808 20:15:02.853231 16599 raft_consensus.cc:482] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:15:02.848639 16597 leader_election.cc:288] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:02.796025 16798 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:02.977998 12244 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:15:04.549451 12045 raft_consensus.cc:2454] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
W20260808 20:15:04.799664 12251 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1933) took 3078 ms (client timeout 2817 ms). Trace:
I20260808 20:15:03.005573 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:15:04.800309 12250 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1938) took 3046 ms (client timeout 1899 ms). Trace:
W20260808 20:15:04.597540 12245 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1943) took 1726 ms (client timeout 1750 ms). Trace:
I20260808 20:15:04.737977 16929 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:04.653833 16701 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:03.356154 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:04.647604 16710 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:04.833699 13077 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 807) took 1478 ms. Trace:
I20260808 20:15:04.654246 16588 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:03.223032 16903 raft_consensus.cc:482] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
W20260808 20:15:04.548256 12000 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.766s user 0.005s sys 0.015s
I20260808 20:15:03.075475 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:04.551659 12047 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:15:04.655664 13072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53296 (request call id 797) took 3812 ms. Trace:
W20260808 20:15:04.552400 12152 rpcz_store.cc:269] 0808 20:15:02.901115 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.901239 (+ 124us) service_pool.cc:227] Handling call
0808 20:15:04.551732 (+1650493us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:15:04.838734 12152 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1849) took 1935 ms (client timeout 1727 ms). Trace:
W20260808 20:15:04.839155 12152 rpcz_store.cc:269] 0808 20:15:02.903045 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:04.838530 (+1935485us) service_pool.cc:214] Skipping call since client already timed out
0808 20:15:04.838652 (+ 122us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 20:15:04.820705 12207 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:15:04.840171 16743 log.cc:924] Time spent T a383bcab83014009b5698c63cdac298b P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 4.727s user 0.028s sys 0.079s
W20260808 20:15:04.657347 12146 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1837) took 2383 ms (client timeout 1899 ms). Trace:
W20260808 20:15:04.549454 13018 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.762s user 0.017s sys 0.022s
I20260808 20:15:04.598059 12043 raft_consensus.cc:2379] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 7: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
W20260808 20:15:04.842131 12043 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 2140) took 2263 ms (client timeout 2092 ms). Trace:
W20260808 20:15:04.830261 12137 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.975s user 0.015s sys 0.020s
W20260808 20:15:04.842731 12043 rpcz_store.cc:269] 0808 20:15:02.578658 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.909508 (+330850us) service_pool.cc:227] Handling call
0808 20:15:04.842048 (+1932540us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:04.829206 13078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 808) took 1469 ms. Trace:
W20260808 20:15:04.734292 13014 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.916s user 0.004s sys 0.012s
W20260808 20:15:04.837167 12045 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1604) took 2178 ms (client timeout 2027 ms). Trace:
W20260808 20:15:04.844532 13014 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.026s user 0.005s sys 0.013s
I20260808 20:15:04.830250 12205 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:15:02.989027 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:02.989223 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:15:04.827742 12040 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1608) took 1971 ms (client timeout 1928 ms). Trace:
W20260808 20:15:04.846139 12249 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1942) took 1975 ms (client timeout 2143 ms). Trace:
W20260808 20:15:04.762422 11995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.960s user 0.016s sys 0.010s
W20260808 20:15:03.819947 11994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.004s sys 0.008s
I20260808 20:15:02.649608 16706 raft_consensus.cc:504] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:15:03.819641 16566 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.516s user 0.001s sys 0.011s
W20260808 20:15:03.513140 15674 consensus_meta.cc:287] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.672s user 0.007s sys 0.003s
I20260808 20:15:04.832916 16875 leader_election.cc:288] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:15:03.500027 16670 consensus_meta.cc:287] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.848s user 0.006s sys 0.001s
I20260808 20:15:02.988620 16400 raft_consensus.cc:2781] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260808 20:15:03.060429 16775 consensus_meta.cc:287] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.974s user 0.009s sys 0.000s
I20260808 20:15:04.846271 12048 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
W20260808 20:15:04.924618 12048 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 2125) took 3325 ms (client timeout 2902 ms). Trace:
I20260808 20:15:04.747956 12150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 467) took 2204 ms. Trace:
I20260808 20:15:04.825523 16675 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:04.641924 16868 raft_consensus.cc:482] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:04.687316 13079 raft_consensus.cc:3037] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
W20260808 20:15:04.899631 12147 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1846) took 1998 ms (client timeout 2012 ms). Trace:
I20260808 20:15:04.843946 16597 raft_consensus.cc:482] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
W20260808 20:15:04.846861 12245 rpcz_store.cc:269] 0808 20:15:02.871492 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.935837 (+ 64345us) service_pool.cc:227] Handling call
0808 20:15:04.597466 (+1661629us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:15:04.899919 11994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.714s user 0.019s sys 0.010s
W20260808 20:15:04.836776 12250 rpcz_store.cc:269] 0808 20:15:01.753710 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.099028 (+345318us) service_pool.cc:227] Handling call
0808 20:15:04.800135 (+2701107us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b5400375854. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:15:04.800238 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":678899584}
W20260808 20:15:04.836548 12251 rpcz_store.cc:269] 0808 20:15:01.720839 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:01.721601 (+ 762us) service_pool.cc:227] Handling call
0808 20:15:04.799342 (+3077741us) spinlock_profiling.cc:243] Waited 233 ms on lock 0x7b5400086e54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:15:04.799477 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":536942848}
W20260808 20:15:04.846519 12249 rpcz_store.cc:269] 0808 20:15:02.871187 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.988680 (+117493us) service_pool.cc:227] Handling call
0808 20:15:04.846082 (+1857402us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:15:04.837648 12047 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 2123) took 3241 ms (client timeout 3273 ms). Trace:
I20260808 20:15:04.859002 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:15:04.840127 12152 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1539) took 1629 ms (client timeout 1503 ms). Trace:
W20260808 20:15:04.846344 12040 rpcz_store.cc:269] 0808 20:15:02.856025 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.943966 (+ 87941us) service_pool.cc:227] Handling call
0808 20:15:04.827666 (+1883700us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:04.844264 13072 rpcz_store.cc:276] 0808 20:15:00.842770 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:00.856699 (+ 13929us) service_pool.cc:227] Handling call
0808 20:15:00.857021 (+ 322us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:01.003223 (+146202us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 20:15:01.460353 (+457130us) log.cc:842] Serialized 14 byte log entry
0808 20:15:01.772749 (+312396us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:01.772798 (+ 49us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:15:01.772845 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:04.655419 (+2882574us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b54004f0454. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:15:04.655467 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:04.655605 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":199150,"prepare.queue_time_us":623079,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":11000,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":211636,"spinlock_wait_cycles":335709952,"thread_start_us":758261,"threads_started":2,"wal-append.queue_time_us":697861,"wal-append.run_cpu_time_us":1789,"wal-append.run_wall_time_us":1391947}
I20260808 20:15:02.935307 16908 raft_consensus.cc:2781] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260808 20:15:04.908017 12045 rpcz_store.cc:269] 0808 20:15:02.658661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.887570 (+228909us) service_pool.cc:227] Handling call
0808 20:15:04.837088 (+1949518us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:15:04.984124 12105 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.202s user 0.002s sys 0.018s
W20260808 20:15:04.909264 12043 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1602) took 2251 ms (client timeout 2249 ms). Trace:
W20260808 20:15:04.987528 12105 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.361s user 0.002s sys 0.018s
W20260808 20:15:04.987773 12043 rpcz_store.cc:269] 0808 20:15:02.658183 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:04.908979 (+2250796us) service_pool.cc:214] Skipping call since client already timed out
0808 20:15:04.909191 (+ 212us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260808 20:15:03.897074 16644 consensus_meta.cc:287] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.954s user 0.009s sys 0.000s
W20260808 20:15:04.991453 12254 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.601s user 0.018s sys 0.019s
I20260808 20:15:02.943943 16855 raft_consensus.cc:2726] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 20:15:02.917027 16892 raft_consensus.cc:2781] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:15:02.918010 16901 raft_consensus.cc:2781] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 20:15:05.012270 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:15:04.845141 12146 rpcz_store.cc:269] 0808 20:15:02.273627 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.397744 (+124117us) service_pool.cc:227] Handling call
0808 20:15:04.657101 (+2259357us) spinlock_profiling.cc:243] Waited 19.4 ms on lock 0x7b540022f6d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:15:04.657206 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":44538240}
I20260808 20:15:04.856814 12137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46680 (request call id 463) took 2301 ms. Trace:
I20260808 20:15:04.843909 13078 rpcz_store.cc:276] 0808 20:15:03.359911 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:03.360065 (+ 154us) service_pool.cc:227] Handling call
0808 20:15:04.829146 (+1469081us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:15:05.018244 12101 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.159s user 0.015s sys 0.008s
W20260808 20:15:05.019806 12101 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.160s user 0.015s sys 0.009s
I20260808 20:15:05.014583 12137 rpcz_store.cc:276] 0808 20:15:02.542234 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.652189 (+109955us) service_pool.cc:227] Handling call
0808 20:15:02.757450 (+105261us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:02.757656 (+ 206us) raft_consensus.cc:1520] Early marking committed up to index 7
0808 20:15:02.855332 (+ 97676us) log.cc:842] Serialized 14 byte log entry
0808 20:15:04.843177 (+1987845us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:04.843230 (+ 53us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:15:04.843277 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:04.843339 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:04.843471 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":131221,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"raft.queue_time_us":81750,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":99,"thread_start_us":1987491,"threads_started":1,"wal-append.queue_time_us":945251}
I20260808 20:15:04.834049 13077 rpcz_store.cc:276] 0808 20:15:03.355707 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:03.355829 (+ 122us) service_pool.cc:227] Handling call
0808 20:15:04.833622 (+1477793us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:15:03.526552 16853 consensus_meta.cc:287] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20: Time spent flushing consensus metadata: real 0.691s user 0.006s sys 0.003s
W20260808 20:15:04.738013 16857 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.407s user 0.003s sys 0.022s
I20260808 20:15:03.059317 16861 raft_consensus.cc:482] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:04.917546 16805 raft_consensus.cc:482] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:15:03.341061 16605 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:04.007961 16938 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:04.655010 16703 raft_consensus.cc:482] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:04.839697 16746 raft_consensus.cc:3037] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
W20260808 20:15:04.633958 16919 log.cc:924] Time spent T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.246s user 0.000s sys 0.000s
I20260808 20:15:02.936046 16867 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:02.965910 16830 raft_consensus.cc:482] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:15:02.936280 16817 raft_consensus.cc:482] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:03.228370 16542 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:15:04.920207 13076 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 803) took 1930 ms (client timeout 1743 ms). Trace:
I20260808 20:15:03.830173 16899 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:04.823670 16802 leader_election.cc:288] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:04.638653 16864 raft_consensus.cc:2726] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260808 20:15:03.820055 16438 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.735s user 0.006s sys 0.006s
I20260808 20:15:03.197378 16661 leader_election.cc:288] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:04.921512 16400 raft_consensus.cc:482] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:15:04.922035 16670 raft_consensus.cc:504] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:04.927508 16775 raft_consensus.cc:504] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:15:03.520555 16508 consensus_meta.cc:287] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.959s user 0.007s sys 0.003s
W20260808 20:15:04.925081 12048 rpcz_store.cc:269] 0808 20:15:01.599253 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.887735 (+1288482us) service_pool.cc:227] Handling call
0808 20:15:04.924538 (+2036803us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:05.154352 16508 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:05.154879 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:04.925637 12150 rpcz_store.cc:276] 0808 20:15:02.543291 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.739332 (+196041us) service_pool.cc:227] Handling call
0808 20:15:02.739669 (+ 337us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:15:02.832114 (+ 92445us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:02.832151 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:15:02.832800 (+ 649us) log.cc:842] Serialized 26 byte log entry
0808 20:15:03.056646 (+223846us) raft_consensus.cc:1631] Marking committed up to 8
0808 20:15:03.213103 (+156457us) log.cc:842] Serialized 14 byte log entry
0808 20:15:03.228516 (+ 15413us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:03.228593 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:15:04.747729 (+1519136us) raft_consensus.cc:1671] finished
0808 20:15:04.747750 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:04.747888 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":106493,"prepare.queue_time_us":156200,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":107141,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":20009,"thread_start_us":223469,"threads_started":1,"wal-append.queue_time_us":1912101}
W20260808 20:15:05.155984 12150 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1535) took 1948 ms (client timeout 1928 ms). Trace:
W20260808 20:15:05.156304 12150 rpcz_store.cc:269] 0808 20:15:03.207992 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.155796 (+1947804us) service_pool.cc:214] Skipping call since client already timed out
0808 20:15:05.155911 (+ 115us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 20:15:05.157164 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:04.950795 16903 raft_consensus.cc:504] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:04.846666 16701 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:04.834826 16588 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:04.935848 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:04.844241 16599 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Advancing to term 10
W20260808 20:15:05.172791 16896 log.cc:924] Time spent T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 2.947s user 0.000s sys 0.045s
I20260808 20:15:05.174767 12046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 2116) took 4355 ms. Trace:
I20260808 20:15:05.175125 12046 rpcz_store.cc:276] 0808 20:15:00.819345 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:00.860199 (+ 40854us) service_pool.cc:227] Handling call
0808 20:15:00.881386 (+ 21187us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:15:00.881697 (+ 311us) raft_consensus.cc:1520] Early marking committed up to index 5
0808 20:15:01.615111 (+733414us) log.cc:842] Serialized 14 byte log entry
0808 20:15:02.127591 (+512480us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:15:02.128044 (+ 453us) log.cc:842] Serialized 26 byte log entry
0808 20:15:02.128160 (+ 116us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:15:02.137723 (+ 9563us) log.cc:842] Serialized 14 byte log entry
0808 20:15:02.357864 (+220141us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:02.357935 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:15:05.172233 (+2814298us) spinlock_profiling.cc:243] Waited 9.67 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:15:05.174533 (+ 2300us) raft_consensus.cc:1671] finished
0808 20:15:05.174554 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:05.174688 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3890,"prepare.queue_time_us":658876,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":77,"raft.queue_time_us":103929,"raft.run_cpu_time_us":357,"raft.run_wall_time_us":21804,"spinlock_wait_cycles":22243456,"thread_start_us":1386413,"threads_started":4,"wal-append.queue_time_us":594768}
W20260808 20:15:04.846662 11995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.045s user 0.017s sys 0.011s
I20260808 20:15:05.177295 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:04.949046 12247 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 636) took 2963 ms. Trace:
I20260808 20:15:05.178038 16712 leader_election.cc:288] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:05.178594 16864 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:05.178834 12244 raft_consensus.cc:595] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:15:05.179199 12244 raft_consensus.cc:603] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 20:15:04.793062 13073 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57330 (request call id 546) took 4005 ms (client timeout 1646 ms). Trace:
W20260808 20:15:05.179629 13073 rpcz_store.cc:269] 0808 20:15:00.788101 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:00.788230 (+ 129us) service_pool.cc:227] Handling call
0808 20:15:04.792788 (+4004558us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54003e5b54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:15:04.792945 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2478583680}
W20260808 20:15:04.985553 11994 connection.cc:232] RPC call timeout handler was delayed by -1.8302988449577242s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19930000000000003s for a response
W20260808 20:15:04.985163 12147 rpcz_store.cc:269] 0808 20:15:02.900816 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.935790 (+ 34974us) service_pool.cc:227] Handling call
0808 20:15:04.899567 (+1963777us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:15:04.985852 12152 rpcz_store.cc:269] 0808 20:15:03.210578 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:04.839954 (+1629376us) service_pool.cc:214] Skipping call since client already timed out
0808 20:15:04.840057 (+ 103us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 20:15:04.008563 16932 raft_consensus.cc:482] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:15:04.802621 16837 log.cc:924] Time spent T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 3.265s user 0.019s sys 0.024s
W20260808 20:15:04.991852 12251 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1951) took 1967 ms (client timeout 1791 ms). Trace:
W20260808 20:15:04.987896 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 1.36161e+06us
W20260808 20:15:04.992430 12250 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2100) took 2079 ms (client timeout 1704 ms). Trace:
I20260808 20:15:05.000002 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:15:04.991941 13072 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 812) took 1622 ms (client timeout 1565 ms). Trace:
W20260808 20:15:04.993660 12245 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1954) took 1819 ms (client timeout 1727 ms). Trace:
W20260808 20:15:04.984817 12047 rpcz_store.cc:269] 0808 20:15:01.596292 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.960291 (+1363999us) service_pool.cc:227] Handling call
0808 20:15:04.837592 (+1877301us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:15:04.994848 12249 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1956) took 1788 ms (client timeout 1602 ms). Trace:
W20260808 20:15:05.009603 12254 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.619s user 0.023s sys 0.019s
I20260808 20:15:04.919516 16334 raft_consensus.cc:3037] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:15:04.852541 16910 raft_consensus.cc:2726] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 20:15:04.964838 16675 raft_consensus.cc:2781] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:15:05.224495 16675 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:05.224866 16675 raft_consensus.cc:3037] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:15:05.225597 16932 raft_consensus.cc:504] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:05.228134 16932 leader_election.cc:288] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:05.078145 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:04.991492 16908 raft_consensus.cc:482] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:15:05.080806 12146 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1538) took 1870 ms (client timeout 1619 ms). Trace:
W20260808 20:15:05.231395 12146 rpcz_store.cc:269] 0808 20:15:03.210020 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.080573 (+1870553us) service_pool.cc:214] Skipping call since client already timed out
0808 20:15:05.080727 (+ 154us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260808 20:15:05.081334 12043 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1609) took 2190 ms (client timeout 2335 ms). Trace:
W20260808 20:15:05.232213 12043 rpcz_store.cc:269] 0808 20:15:02.891216 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:04.988485 (+2097269us) service_pool.cc:227] Handling call
0808 20:15:05.081268 (+ 92783us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:04.920665 15674 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:05.111075 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:05.111809 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:05.112628 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:05.012647 16901 raft_consensus.cc:482] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:15:05.012796 16892 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
W20260808 20:15:05.109056 16857 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.968s user 0.011s sys 0.025s
I20260808 20:15:04.831993 16929 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:05.253022 16929 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:05.000993 16644 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:05.254741 16644 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:04.999702 16868 raft_consensus.cc:504] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:05.255695 16599 raft_consensus.cc:504] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:05.256613 16868 leader_election.cc:288] T a383bcab83014009b5698c63cdac298b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:15:05.257580 16918 log.cc:924] Time spent T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260808 20:15:05.257797 16599 leader_election.cc:288] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 10 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:05.258044 13074 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
W20260808 20:15:05.125614 13076 rpcz_store.cc:269] 0808 20:15:02.989651 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.989995 (+ 344us) service_pool.cc:227] Handling call
0808 20:15:04.920137 (+1930142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:05.259148 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:05.259765 13076 raft_consensus.cc:2454] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:15:05.260197 13076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 809) took 1900 ms. Trace:
I20260808 20:15:05.260458 13076 rpcz_store.cc:276] 0808 20:15:03.360251 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.258846 (+1898595us) service_pool.cc:227] Handling call
0808 20:15:05.260147 (+ 1301us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:05.261276 12041 raft_consensus.cc:2379] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 10: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:15:05.149103 16400 raft_consensus.cc:3037] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:15:05.108247 16853 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:05.180781 16701 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:05.265455 13075 raft_consensus.cc:1226] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 4. Current term is 5. Ops: []
I20260808 20:15:05.266669 16853 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:05.160826 12038 raft_consensus.cc:2379] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 9: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:15:04.930546 16798 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:15:05.149979 16438 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.073s user 0.014s sys 0.015s
I20260808 20:15:05.113641 16861 raft_consensus.cc:504] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:15:05.168258 12150 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1848) took 2266 ms (client timeout 2295 ms). Trace:
I20260808 20:15:05.169409 12248 raft_consensus.cc:2454] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:15:05.179121 16661 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
W20260808 20:15:04.954847 13015 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.134s user 0.021s sys 0.021s
W20260808 20:15:05.181836 11994 connection.cc:232] RPC call timeout handler was delayed by -1.4751726689282805s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.236s for a response
W20260808 20:15:05.269656 12248 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1929) took 3576 ms (client timeout 2130 ms). Trace:
W20260808 20:15:05.270102 11994 connection.cc:232] RPC call timeout handler was delayed by -1.4475276577286422s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2222s for a response
W20260808 20:15:05.270366 12248 rpcz_store.cc:269] 0808 20:15:01.693541 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:01.715669 (+ 22128us) service_pool.cc:227] Handling call
0808 20:15:04.935691 (+3220022us) spinlock_profiling.cc:243] Waited 1.18 ms on lock 0x7b58001a1398. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537ab7c 00007fa96537a73b 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:15:05.269589 (+333898us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2725504}
I20260808 20:15:05.269760 16661 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:15:05.271586 12248 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1955) took 2065 ms (client timeout 1717 ms). Trace:
W20260808 20:15:05.271948 12248 rpcz_store.cc:269] 0808 20:15:03.206348 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.271378 (+2065030us) service_pool.cc:214] Skipping call since client already timed out
0808 20:15:05.271498 (+ 120us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260808 20:15:05.273149 12248 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1950) took 2249 ms (client timeout 2012 ms). Trace:
W20260808 20:15:05.273473 12248 rpcz_store.cc:269] 0808 20:15:03.023827 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.272586 (+2248759us) service_pool.cc:214] Skipping call since client already timed out
0808 20:15:05.273075 (+ 489us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260808 20:15:05.182228 12203 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53030 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 2 similar messages]
W20260808 20:15:05.274252 12248 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1946) took 2365 ms (client timeout 2130 ms). Trace:
W20260808 20:15:05.269965 13015 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.449s user 0.022s sys 0.026s
I20260808 20:15:05.180681 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:05.276505 12040 raft_consensus.cc:2379] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 7: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:15:05.119890 16938 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:15:05.220072 12245 rpcz_store.cc:269] 0808 20:15:03.173846 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:04.993180 (+1819334us) service_pool.cc:214] Skipping call since client already timed out
0808 20:15:04.993513 (+ 333us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 20:15:05.189041 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:15:05.221375 12251 rpcz_store.cc:269] 0808 20:15:03.024147 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:04.991670 (+1967523us) service_pool.cc:214] Skipping call since client already timed out
0808 20:15:04.991788 (+ 118us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260808 20:15:05.221938 12250 rpcz_store.cc:269] 0808 20:15:02.913205 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:04.992250 (+2079045us) service_pool.cc:214] Skipping call since client already timed out
0808 20:15:04.992364 (+ 114us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 20:15:05.181950 16779 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
W20260808 20:15:05.220734 12249 rpcz_store.cc:269] 0808 20:15:03.206672 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:04.994678 (+1788006us) service_pool.cc:214] Skipping call since client already timed out
0808 20:15:04.994785 (+ 107us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260808 20:15:05.221114 13072 rpcz_store.cc:269] 0808 20:15:03.363336 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:04.985214 (+1621878us) service_pool.cc:214] Skipping call since client already timed out
0808 20:15:04.985326 (+ 112us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 20:15:05.223560 16945 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:05.223242 16864 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:05.221860 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:05.124150 16703 raft_consensus.cc:504] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:05.126396 16830 raft_consensus.cc:504] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:05.235898 16908 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:05.220373 12105 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:05.247687 16901 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Advancing to term 9
W20260808 20:15:05.293123 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 72821.2us
I20260808 20:15:05.234719 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:05.295578 12147 raft_consensus.cc:2396] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260808 20:15:05.130710 16817 raft_consensus.cc:504] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:05.131016 16805 raft_consensus.cc:504] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:15:05.236476 13077 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 813) took 1872 ms (client timeout 2434 ms). Trace:
I20260808 20:15:05.130872 16542 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:15:05.247511 12137 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1537) took 2038 ms (client timeout 1879 ms). Trace:
I20260808 20:15:05.251295 16892 raft_consensus.cc:3037] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:05.269174 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:15:05.305630 12137 rpcz_store.cc:269] 0808 20:15:03.209095 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.081864 (+1872769us) service_pool.cc:227] Handling call
0808 20:15:05.247438 (+165574us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:15:05.269457 12150 rpcz_store.cc:269] 0808 20:15:02.901379 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.156908 (+2255529us) service_pool.cc:227] Handling call
0808 20:15:05.168196 (+ 11288us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:05.282786 16853 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:15:05.270630 11994 connection.cc:232] RPC call timeout handler was delayed by -1.3586900730151683s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15610000000000002s for a response
W20260808 20:15:05.274616 12248 rpcz_store.cc:269] 0808 20:15:02.909246 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.274098 (+2364852us) service_pool.cc:214] Skipping call since client already timed out
0808 20:15:05.274202 (+ 104us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 20:15:05.277539 13080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53296 (request call id 801) took 2623 ms. Trace:
I20260808 20:15:05.283450 16508 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:15:05.281478 13078 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 810) took 1919 ms (client timeout 1693 ms). Trace:
W20260808 20:15:05.313941 13078 rpcz_store.cc:269] 0808 20:15:03.362753 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.016075 (+1653322us) service_pool.cc:227] Handling call
0808 20:15:05.281393 (+265318us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:04.995759 16597 raft_consensus.cc:504] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:05.286365 12152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1534) took 2079 ms. Trace:
I20260808 20:15:05.123870 16867 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:05.274806 12203 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53030 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1949)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2099)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1953)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1948)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2116)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2117)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2107)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2120)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2124)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2125)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2110)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2113)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2112)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2114)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2122)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2102)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2126)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2111)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2119)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2108)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2121)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2123)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53030 (request call id 646)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2095)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2096)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2097)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2098)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1947)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1952)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2101)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2103)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2104)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2105)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2106)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2109)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2115)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2118)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 642)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 643)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 644)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 645)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 647)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 648)
I20260808 20:15:05.319480 16867 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:05.321012 16542 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:05.323422 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:05.325323 13076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53296 (request call id 814) took 1961 ms. Trace:
I20260808 20:15:05.325660 13076 rpcz_store.cc:276] 0808 20:15:03.363876 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.324322 (+1960446us) service_pool.cc:227] Handling call
0808 20:15:05.324815 (+ 493us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:05.325018 (+ 203us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:05.325048 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:05.325087 (+ 39us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:15:05.325121 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:05.325176 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:05.325273 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:05.307812 13080 rpcz_store.cc:276] 0808 20:15:02.654189 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.654328 (+ 139us) service_pool.cc:227] Handling call
0808 20:15:02.654647 (+ 319us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:15:02.654942 (+ 295us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:02.654967 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:15:02.670999 (+ 16032us) log.cc:842] Serialized 26 byte log entry
0808 20:15:03.194337 (+523338us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:15:03.194379 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:03.194459 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:15:05.277311 (+2082852us) raft_consensus.cc:1671] finished
0808 20:15:05.277332 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:05.277485 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26664,"raft.queue_time_us":89,"raft.run_cpu_time_us":4085,"raft.run_wall_time_us":63009,"thread_start_us":259429,"threads_started":1,"wal-append.queue_time_us":2310642}
I20260808 20:15:05.287647 12047 raft_consensus.cc:2454] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:15:05.269129 16438 raft_consensus.cc:686] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:15:05.229399 12244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:40912 (request call id 30) took 2434 ms. Trace:
I20260808 20:15:05.178363 12247 rpcz_store.cc:276] 0808 20:15:01.985454 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.648088 (+662634us) service_pool.cc:227] Handling call
0808 20:15:02.648369 (+ 281us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:02.650332 (+ 1963us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:02.650359 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:02.650395 (+ 36us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:15:02.650428 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:04.948761 (+2298333us) spinlock_profiling.cc:243] Waited 1.78 ms on lock 0x7b54000d6e54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:15:04.948821 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:04.948985 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4083072}
I20260808 20:15:05.290755 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:05.278501 16649 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:05.296826 16400 raft_consensus.cc:504] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:05.303823 16864 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:05.347429 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:15:05.302258 12147 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53162 (request call id 1536) took 2093 ms (client timeout 2335 ms). Trace:
I20260808 20:15:05.303164 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:05.276657 13073 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 811) took 1913 ms. Trace:
W20260808 20:15:05.277292 13015 connection.cc:232] RPC call timeout handler was delayed by -1.1989352230448276s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1524s for a response
W20260808 20:15:05.305348 13077 rpcz_store.cc:269] 0808 20:15:03.363601 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.055095 (+1691494us) service_pool.cc:227] Handling call
0808 20:15:05.236387 (+181292us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:15:05.307657 11994 connection.cc:232] RPC call timeout handler was delayed by -1.3455292310100049s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1678s for a response
I20260808 20:15:05.348666 13073 rpcz_store.cc:276] 0808 20:15:03.363057 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.180423 (+1817366us) service_pool.cc:227] Handling call
0808 20:15:05.276596 (+ 96173us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:15:05.349191 11994 connection.cc:232] RPC call timeout handler was delayed by -1.3320540019776672s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17500000000000002s for a response
W20260808 20:15:05.349524 11994 connection.cc:232] RPC call timeout handler was delayed by -1.2895953317638487s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2817s for a response
W20260808 20:15:05.349852 11994 connection.cc:232] RPC call timeout handler was delayed by -1.0772494138218462s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2143s for a response
I20260808 20:15:05.309842 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 2115) took 4499 ms. Trace:
I20260808 20:15:05.316506 12152 rpcz_store.cc:276] 0808 20:15:03.207383 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.188748 (+1981365us) service_pool.cc:227] Handling call
0808 20:15:05.286273 (+ 97525us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:15:05.334035 12047 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1611) took 1533 ms (client timeout 1619 ms). Trace:
W20260808 20:15:05.350814 13015 connection.cc:232] RPC call timeout handler was delayed by -1.1129242971073836s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2705s for a response
I20260808 20:15:05.350301 12039 rpcz_store.cc:276] 0808 20:15:00.810567 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:00.812791 (+ 2224us) service_pool.cc:227] Handling call
0808 20:15:00.842718 (+ 29927us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:15:00.851686 (+ 8968us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:00.851716 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:15:00.852088 (+ 372us) log.cc:842] Serialized 26 byte log entry
0808 20:15:01.254611 (+402523us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:15:01.254651 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:01.254719 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:15:04.800722 (+3546003us) spinlock_profiling.cc:243] Waited 28.5 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:15:05.306084 (+505362us) raft_consensus.cc:1671] finished
0808 20:15:05.306104 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:05.309781 (+ 3677us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8642,"raft.queue_time_us":34163,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":4582,"spinlock_wait_cycles":65448704,"thread_start_us":370716,"threads_started":1,"wal-append.queue_time_us":642161}
I20260808 20:15:05.351720 13015 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
W20260808 20:15:05.348256 12147 rpcz_store.cc:269] 0808 20:15:03.208539 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.189102 (+1980563us) service_pool.cc:227] Handling call
0808 20:15:05.302197 (+113095us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:15:05.350728 12047 rpcz_store.cc:269] 0808 20:15:03.801067 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.221580 (+1420513us) service_pool.cc:227] Handling call
0808 20:15:05.333970 (+112390us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:05.353542 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:05.335776 13079 raft_consensus.cc:2454] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:15:05.339284 12244 rpcz_store.cc:276] 0808 20:15:02.757487 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.894143 (+136656us) service_pool.cc:227] Handling call
0808 20:15:05.191290 (+2297147us) spinlock_profiling.cc:243] Waited 351 ms on lock 0x7b5400087354. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682d4bc3 00007fa96d21e81a 00007fa966911d04 00007fa966911c92 00007fa966911bc1 00007fa966911b3c 00007fa966910df2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:15:05.191409 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":806915328}
I20260808 20:15:05.348248 12048 raft_consensus.cc:2454] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 3.
W20260808 20:15:05.342734 12105 connection.cc:232] RPC call timeout handler was delayed by -1.6419688030146062s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1743s for a response
I20260808 20:15:05.340296 12043 raft_consensus.cc:2454] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
W20260808 20:15:05.355597 12043 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 2132) took 3088 ms (client timeout 3180 ms). Trace:
I20260808 20:15:05.301637 16765 raft_consensus.cc:2726] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 20:15:05.298599 16830 leader_election.cc:288] T efc388dadfe84be190c9ddd0b3725244 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:05.359846 11994 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:05.366982 11994 leader_election.cc:302] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:15:05.355278 12105 connection.cc:232] RPC call timeout handler was delayed by -1.362068803049624s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2054s for a response
I20260808 20:15:05.355829 13015 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters: b9d94f35ea15421aa1d721201c782d8a
W20260808 20:15:05.368040 12105 connection.cc:232] RPC call timeout handler was delayed by -1.3316769890952855s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1565s for a response
W20260808 20:15:05.368357 12105 connection.cc:232] RPC call timeout handler was delayed by -1.3233688028994948s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2097s for a response
W20260808 20:15:05.357522 12043 rpcz_store.cc:269] 0808 20:15:02.267210 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.250554 (+2983344us) service_pool.cc:227] Handling call
0808 20:15:05.355541 (+104987us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:05.331962 16706 leader_election.cc:288] T ddb83ac6240244be967995e7771a8b64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:05.363319 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:05.371443 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:05.354395 12047 raft_consensus.cc:2454] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:15:05.370740 11994 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
W20260808 20:15:05.365114 12103 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.623s user 0.008s sys 0.035s
W20260808 20:15:05.374346 12103 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.111s user 0.009s sys 0.035s
W20260808 20:15:05.376915 13071 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 806) took 2021 ms (client timeout 2054 ms). Trace:
I20260808 20:15:05.376206 11994 leader_election.cc:302] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:05.378738 11994 leader_election.cc:302] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:15:05.379416 12105 connection.cc:232] RPC call timeout handler was delayed by -1.2479609579313546s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1693s for a response
I20260808 20:15:05.302899 16779 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:15:05.377303 13071 rpcz_store.cc:269] 0808 20:15:03.355314 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.371198 (+2015884us) service_pool.cc:227] Handling call
0808 20:15:05.376839 (+ 5641us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:05.390702 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:05.394745 12244 raft_consensus.cc:2396] T c4dc15f09b4a4470ac96054b70602187 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 6 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260808 20:15:05.391196 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:05.396787 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:15:05.402220 12038 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1601) took 3018 ms (client timeout 1676 ms). Trace:
W20260808 20:15:05.402608 12038 rpcz_store.cc:269] 0808 20:15:02.383704 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.653714 (+270010us) service_pool.cc:227] Handling call
0808 20:15:05.401965 (+2748251us) spinlock_profiling.cc:243] Waited 2.31 s on lock 0x7b5400464bd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:15:05.402101 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5303033600}
I20260808 20:15:05.403580 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:15:05.404282 12038 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1600) took 3025 ms (client timeout 3323 ms). Trace:
I20260808 20:15:05.409360 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:05.410202 12247 raft_consensus.cc:2396] T e5dc2ab0712f4054883178f4b4c50928 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260808 20:15:05.411235 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:05.411891 12247 raft_consensus.cc:2354] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6.
I20260808 20:15:05.412577 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:05.421787 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:05.422493 12247 raft_consensus.cc:2454] T 1d7d7a85e3964f3f9d5f2dc39e23340b P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:15:05.423357 16903 leader_election.cc:288] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:05.423760 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:05.424470 12247 raft_consensus.cc:2454] T e256c29dec964f49856cf204f950793b P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 7.
I20260808 20:15:05.425678 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:05.426321 12247 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 9.
I20260808 20:15:05.428712 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:05.429387 16861 leader_election.cc:288] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:05.434371 16805 leader_election.cc:288] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:15:05.435801 12102 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.635s user 0.070s sys 0.163s
W20260808 20:15:05.436419 12102 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.636s user 0.070s sys 0.163s
I20260808 20:15:05.433382 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:05.438225 12247 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 7.
I20260808 20:15:05.395790 12039 raft_consensus.cc:2454] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
W20260808 20:15:05.450579 12039 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1613) took 1601 ms (client timeout 1634 ms). Trace:
W20260808 20:15:05.450915 12206 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.846s user 0.020s sys 0.055s
W20260808 20:15:05.451925 12206 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.847s user 0.020s sys 0.055s
W20260808 20:15:05.404587 12038 rpcz_store.cc:269] 0808 20:15:02.378576 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.403335 (+3024759us) service_pool.cc:227] Handling call
0808 20:15:05.404233 (+ 898us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:05.452562 12206 leader_election.cc:302] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:15:05.453699 12206 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
W20260808 20:15:05.451273 12039 rpcz_store.cc:269] 0808 20:15:03.849543 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.363043 (+1513500us) service_pool.cc:227] Handling call
0808 20:15:05.450496 (+ 87453us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:05.457857 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:05.458703 12043 raft_consensus.cc:2396] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260808 20:15:05.364936 16857 raft_consensus.cc:2781] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 20:15:05.305114 16945 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:05.364020 16864 raft_consensus.cc:482] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:15:05.337836 16853 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:05.360386 16661 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:15:05.475380 12102 connection.cc:232] RPC call timeout handler was delayed by -1.5909855659119785s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21389999999999998s for a response
I20260808 20:15:05.431596 16588 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:15:05.492079 12043 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 2138) took 2951 ms (client timeout 3155 ms). Trace:
I20260808 20:15:05.373519 16508 raft_consensus.cc:2781] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:15:05.421846 16703 leader_election.cc:288] T e020c14c3aeb419493cd543071ed7801 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:05.355306 16649 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:05.499625 12206 leader_election.cc:302] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:15:05.503526 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:15:05.507347 12102 connection.cc:232] RPC call timeout handler was delayed by -1.4751049980986863s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1565s for a response
W20260808 20:15:05.935057 12141 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1821) took 5547 ms (client timeout 3003 ms). Trace:
W20260808 20:15:06.699862 12102 connection.cc:232] RPC call timeout handler was delayed by -1.2297724129166454s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2902s for a response
W20260808 20:15:06.700260 12102 connection.cc:232] RPC call timeout handler was delayed by -1.0008049979805946s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20920000000000002s for a response
I20260808 20:15:06.700790 12102 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
W20260808 20:15:05.498251 12047 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1607) took 2713 ms (client timeout 2846 ms). Trace:
W20260808 20:15:06.703620 12141 rpcz_store.cc:269] 0808 20:15:00.387152 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:00.387264 (+ 112us) service_pool.cc:227] Handling call
0808 20:15:05.934819 (+5547555us) spinlock_profiling.cc:243] Waited 2.06 s on lock 0x7b540017fa54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:15:05.934963 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4732899200}
W20260808 20:15:06.704195 12047 rpcz_store.cc:269] 0808 20:15:02.785071 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.353282 (+2568211us) service_pool.cc:227] Handling call
0808 20:15:05.498187 (+144905us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:05.484812 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:06.706082 12244 raft_consensus.cc:2396] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for term 11 because replica has last-logged OpId of term: 10 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 10.
W20260808 20:15:06.706523 12244 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2102) took 2858 ms (client timeout 3119 ms). Trace:
W20260808 20:15:06.706844 12244 rpcz_store.cc:269] 0808 20:15:03.848361 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.484532 (+1636171us) service_pool.cc:227] Handling call
0808 20:15:06.706466 (+1221934us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:06.707751 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:05.501688 16706 raft_consensus.cc:2781] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 20:15:05.391352 16599 raft_consensus.cc:2781] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260808 20:15:05.612982 16788 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.895s user 0.069s sys 0.145s
W20260808 20:15:06.713444 16788 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 5.002s user 0.083s sys 0.191s
I20260808 20:15:05.383859 16765 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:06.715440 16765 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:15:05.373693 16762 raft_consensus.cc:2781] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 20:15:06.717589 16762 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:15:06.717957 16762 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:05.383528 16830 raft_consensus.cc:2781] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:15:06.721675 16830 raft_consensus.cc:482] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:06.722077 16830 raft_consensus.cc:3037] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:15:05.456857 16868 raft_consensus.cc:2726] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 20:15:05.854267 16892 consensus_meta.cc:287] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.533s user 0.005s sys 0.004s
I20260808 20:15:06.727335 16892 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:05.502416 16857 raft_consensus.cc:482] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:15:06.731132 16857 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:05.446605 16867 raft_consensus.cc:2726] T 72e59eff611e4cc693407979ebe0857c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:15:05.422801 16817 leader_election.cc:288] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:05.500461 16861 raft_consensus.cc:482] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
W20260808 20:15:05.553517 16921 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.794s user 0.030s sys 0.064s
W20260808 20:15:06.736696 16921 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 1.985s user 0.048s sys 0.119s
I20260808 20:15:05.481500 16903 raft_consensus.cc:2781] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election won for term 6
W20260808 20:15:05.922098 16908 consensus_meta.cc:287] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.614s user 0.009s sys 0.001s
I20260808 20:15:06.003479 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1850) took 3100 ms. Trace:
I20260808 20:15:06.738566 16908 raft_consensus.cc:504] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:06.738921 12146 rpcz_store.cc:276] 0808 20:15:02.903344 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.232051 (+2328707us) service_pool.cc:227] Handling call
0808 20:15:05.238272 (+ 6221us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 20:15:05.238647 (+ 375us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:05.238675 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 20:15:05.239179 (+ 504us) log.cc:842] Serialized 52 byte log entry
0808 20:15:05.800022 (+560843us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:15:05.800063 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:05.800182 (+ 119us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:15:06.003224 (+203042us) raft_consensus.cc:1671] finished
0808 20:15:06.003246 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:06.003410 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":744,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":35238,"thread_start_us":490351,"threads_started":1,"wal-append.queue_time_us":710399}
I20260808 20:15:06.739856 12045 raft_consensus.cc:2379] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 9: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:15:06.740437 16908 leader_election.cc:288] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:15:06.254936 12000 leader_election.cc:334] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 3.003s (SENT)
W20260808 20:15:06.741400 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 486480us
W20260808 20:15:06.741932 12045 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 2142) took 4162 ms (client timeout 2434 ms). Trace:
W20260808 20:15:06.742316 12045 rpcz_store.cc:269] 0808 20:15:02.579193 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.000496 (+2421303us) service_pool.cc:227] Handling call
0808 20:15:06.741731 (+1741235us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b540034ec54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:15:06.741866 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3474629888}
W20260808 20:15:06.259910 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 791521us
W20260808 20:15:06.307839 12250 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2116) took 1763 ms (client timeout 1620 ms). Trace:
W20260808 20:15:06.743880 12250 rpcz_store.cc:269] 0808 20:15:04.544379 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.307638 (+1763259us) service_pool.cc:214] Skipping call since client already timed out
0808 20:15:06.307771 (+ 133us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260808 20:15:06.744591 11994 leader_election.cc:334] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Timed out: RequestConsensusVote RPC to 127.10.24.3:38391 timed out after 2.130s (SENT)
I20260808 20:15:06.744720 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:06.317006 16859 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Advancing to term 7
W20260808 20:15:06.338614 12254 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.105s user 0.021s sys 0.021s
I20260808 20:15:06.745036 11994 leader_election.cc:302] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:15:06.745141 16903 raft_consensus.cc:686] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
W20260808 20:15:06.745402 12254 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.525s user 0.023s sys 0.023s
I20260808 20:15:06.745708 16817 raft_consensus.cc:482] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
W20260808 20:15:06.346166 12205 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.500s user 0.003s sys 0.020s
W20260808 20:15:06.746435 12205 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.901s user 0.004s sys 0.021s
I20260808 20:15:06.746084 16903 consensus_queue.cc:237] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:15:06.746713 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 1.91643e+06us
I20260808 20:15:06.747305 12205 leader_election.cc:302] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:15:06.748358 12205 leader_election.cc:302] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:15:05.538597 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
W20260808 20:15:05.559199 13016 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.011s user 0.045s sys 0.077s
W20260808 20:15:06.749670 13016 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.201s user 0.064s sys 0.127s
W20260808 20:15:06.750622 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 48074.3us
I20260808 20:15:06.751240 12102 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:15:06.751292 13016 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:15:06.752411 13016 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:15:05.727258 12144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53176 (request call id 1845) took 2826 ms. Trace:
I20260808 20:15:06.752792 16867 raft_consensus.cc:2726] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:15:06.753134 16942 raft_consensus.cc:2726] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:15:06.753397 16867 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
W20260808 20:15:05.692318 12140 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1839) took 3418 ms (client timeout 1819 ms). Trace:
W20260808 20:15:06.753870 12140 rpcz_store.cc:269] 0808 20:15:02.274117 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.390208 (+116091us) service_pool.cc:227] Handling call
0808 20:15:05.692061 (+3301853us) spinlock_profiling.cc:243] Waited 45.8 ms on lock 0x7b54003ed354. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:15:05.692239 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":105434368}
I20260808 20:15:06.754750 16867 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:06.756641 16867 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:15:05.817737 12249 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1948) took 2875 ms (client timeout 2619 ms). Trace:
W20260808 20:15:06.769465 12249 rpcz_store.cc:269] 0808 20:15:02.942763 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.817566 (+2874803us) service_pool.cc:214] Skipping call since client already timed out
0808 20:15:05.817674 (+ 108us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 20:15:06.770437 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:06.770815 16991 raft_consensus.cc:2781] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:15:06.771111 12249 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:15:06.771652 12249 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2108) took 2229 ms. Trace:
I20260808 20:15:06.771687 12254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2094) took 4119 ms. Trace:
I20260808 20:15:05.609839 15674 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:06.395831 16701 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:06.417513 16597 leader_election.cc:288] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:05.523288 16864 raft_consensus.cc:504] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:06.772168 12254 rpcz_store.cc:276] 0808 20:15:02.652536 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.888457 (+235921us) service_pool.cc:227] Handling call
0808 20:15:02.914507 (+ 26050us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:02.914718 (+ 211us) raft_consensus.cc:1520] Early marking committed up to index 7
0808 20:15:03.390613 (+475895us) log.cc:842] Serialized 14 byte log entry
0808 20:15:06.745752 (+3355139us) log.cc:842] Serialized 14 byte log entry
0808 20:15:06.766823 (+ 21071us) log.cc:842] Serialized 14 byte log entry
0808 20:15:06.771365 (+ 4542us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:06.771416 (+ 51us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:15:06.771454 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:06.771536 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:06.771638 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":2156033,"prepare.run_cpu_time_us":115,"prepare.run_wall_time_us":120,"thread_start_us":3781224,"threads_started":5,"wal-append.queue_time_us":2184953,"wal-append.run_cpu_time_us":2744,"wal-append.run_wall_time_us":1069901}
I20260808 20:15:06.774119 16864 leader_election.cc:288] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:15:06.422797 13074 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53296 (request call id 805) took 3141 ms (client timeout 3273 ms). Trace:
I20260808 20:15:06.425168 16982 raft_consensus.cc:482] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:06.774374 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:15:05.851354 16675 consensus_meta.cc:287] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a: Time spent flushing consensus metadata: real 0.626s user 0.005s sys 0.005s
I20260808 20:15:05.543749 16670 leader_election.cc:288] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:06.437633 16508 raft_consensus.cc:482] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:05.627049 16853 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:06.774857 16982 raft_consensus.cc:504] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:06.772058 12249 rpcz_store.cc:276] 0808 20:15:04.542040 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.770156 (+2228116us) service_pool.cc:227] Handling call
0808 20:15:06.771602 (+ 1446us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:06.776163 16588 raft_consensus.cc:2781] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:15:06.776588 16588 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 20:15:05.817103 12245 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1949) took 2835 ms (client timeout 2130 ms). Trace:
I20260808 20:15:05.862747 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:06.772651 16991 raft_consensus.cc:482] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:06.777385 12138 raft_consensus.cc:2454] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
W20260808 20:15:06.777872 12138 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46680 (request call id 469) took 4234 ms (client timeout 1944 ms). Trace:
W20260808 20:15:06.778190 12138 rpcz_store.cc:269] 0808 20:15:02.543836 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.626088 (+ 82252us) service_pool.cc:227] Handling call
0808 20:15:05.862561 (+3236473us) spinlock_profiling.cc:243] Waited 3.41 ms on lock 0x7b5800004598. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d21c1b0 00007fa96d21ba94 00007fa966903c04 00007fa966903b92
0808 20:15:06.777810 (+915249us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7846272}
I20260808 20:15:05.934049 12243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1930) took 4218 ms. Trace:
I20260808 20:15:06.778987 12243 rpcz_store.cc:276] 0808 20:15:01.719025 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:01.731819 (+ 12794us) service_pool.cc:227] Handling call
0808 20:15:04.787746 (+3055927us) spinlock_profiling.cc:243] Waited 18.1 ms on lock 0x7b58001a1398. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa96537a851 00007fa96537a702 00007fa96e26ab09 00007fa96d1a6730 00007fa96d26c79b 00007fa96d26c821 00007fa96d2302b0 00007fa96d21b7f9 00007fa96d21b2a2 00007fa966901684
0808 20:15:04.787984 (+ 238us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:05.190413 (+402429us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:05.190452 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:05.190492 (+ 40us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:15:05.190520 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:05.933768 (+743248us) spinlock_profiling.cc:243] Waited 402 ms on lock 0x7b5400143054. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:15:05.933821 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:05.933961 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":966703872}
I20260808 20:15:06.780046 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:06.780524 12243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53030 (request call id 646) took 1848 ms. Trace:
W20260808 20:15:05.725303 12048 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1612) took 1895 ms (client timeout 1503 ms). Trace:
W20260808 20:15:05.803251 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 484048us
W20260808 20:15:06.777037 12245 rpcz_store.cc:269] 0808 20:15:02.981627 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.816932 (+2835305us) service_pool.cc:214] Skipping call since client already timed out
0808 20:15:05.817038 (+ 106us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260808 20:15:06.774621 13074 rpcz_store.cc:269] 0808 20:15:03.281521 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:04.858477 (+1576956us) service_pool.cc:227] Handling call
0808 20:15:06.422619 (+1564142us) spinlock_profiling.cc:243] Waited 272 ms on lock 0x7b54005b4854. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:15:06.422737 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":625923456}
W20260808 20:15:06.781116 12048 rpcz_store.cc:269] 0808 20:15:03.830363 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.154497 (+1324134us) service_pool.cc:227] Handling call
0808 20:15:05.725074 (+570577us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b54004858d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:15:05.725207 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":443129088}
I20260808 20:15:06.776922 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:05.691433 12241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 628) took 4022 ms. Trace:
I20260808 20:15:06.783773 12241 rpcz_store.cc:276] 0808 20:15:01.668598 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:01.668729 (+ 131us) service_pool.cc:227] Handling call
0808 20:15:01.669043 (+ 314us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:02.228838 (+559795us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:02.228865 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:02.228932 (+ 67us) raft_consensus.cc:1631] Marking committed up to 8
0808 20:15:02.228972 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:05.691165 (+3462193us) spinlock_profiling.cc:243] Waited 560 ms on lock 0x7b5400142b54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:15:05.691222 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:05.691366 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1286984064}
W20260808 20:15:05.616297 12001 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.069s user 0.012s sys 0.058s
W20260808 20:15:05.655225 12101 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.633s user 0.017s sys 0.009s
I20260808 20:15:05.522318 16775 leader_election.cc:288] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:06.599758 13076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:52008 (request call id 3) took 1687 ms. Trace:
I20260808 20:15:05.890745 16938 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:06.611222 16971 raft_consensus.cc:482] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:05.628508 16945 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:15:06.622586 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 1.12313e+06us
I20260808 20:15:06.624751 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 2146) took 1187 ms. Trace:
I20260808 20:15:06.775570 16508 raft_consensus.cc:3037] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
W20260808 20:15:05.889791 12041 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 2127) took 3963 ms (client timeout 2139 ms). Trace:
W20260808 20:15:05.679242 12040 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 2135) took 3209 ms (client timeout 2853 ms). Trace:
I20260808 20:15:06.634500 16829 raft_consensus.cc:2781] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:15:05.893023 13075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53296 (request call id 804) took 2784 ms. Trace:
I20260808 20:15:05.508846 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:15:06.646793 12142 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1842) took 4371 ms (client timeout 2143 ms). Trace:
I20260808 20:15:05.881672 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 2118) took 4948 ms. Trace:
I20260808 20:15:05.474773 16438 consensus_queue.cc:237] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:15:06.791038 12249 raft_consensus.cc:2379] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 9: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:15:06.831874 16829 raft_consensus.cc:482] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:15:06.832233 12249 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2123) took 2013 ms. Trace:
I20260808 20:15:06.832358 16829 raft_consensus.cc:3037] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:06.832614 12249 rpcz_store.cc:276] 0808 20:15:04.819215 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.776509 (+1957294us) service_pool.cc:227] Handling call
0808 20:15:06.832173 (+ 55664us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:06.833623 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:06.834219 12249 raft_consensus.cc:3037] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 5
I20260808 20:15:05.686934 16779 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:06.834633 16857 raft_consensus.cc:504] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:06.836249 16982 leader_election.cc:288] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:06.836956 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:06.838640 16830 raft_consensus.cc:504] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:06.840510 16830 leader_election.cc:288] T b7e4c0cacf074dee848eed1ab4d27ebf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:15:06.671234 16901 consensus_meta.cc:287] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 1.358s user 0.012s sys 0.000s
I20260808 20:15:06.840746 16829 raft_consensus.cc:504] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:06.841917 12250 raft_consensus.cc:2379] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 7: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
W20260808 20:15:06.842561 12247 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2126) took 2022 ms (client timeout 2492 ms). Trace:
W20260808 20:15:06.842851 12250 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2119) took 2202 ms (client timeout 2853 ms). Trace:
W20260808 20:15:06.843329 12250 rpcz_store.cc:269] 0808 20:15:04.640703 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.744490 (+2103787us) service_pool.cc:227] Handling call
0808 20:15:06.842687 (+ 98197us) spinlock_profiling.cc:243] Waited 96.2 ms on lock 0x7b540029ed54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:15:06.842803 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":221305728}
I20260808 20:15:06.844313 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:15:05.814847 12248 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2099) took 2959 ms (client timeout 2492 ms). Trace:
I20260808 20:15:06.843959 12036 rpcz_store.cc:276] 0808 20:15:00.933717 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:00.933844 (+ 127us) service_pool.cc:227] Handling call
0808 20:15:00.934112 (+ 268us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:01.434182 (+500070us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:01.434213 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:01.434250 (+ 37us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:15:01.434277 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:05.881397 (+4447120us) spinlock_profiling.cc:243] Waited 500 ms on lock 0x7b54003b5e54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:15:05.881450 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:05.881607 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1149641088}
I20260808 20:15:06.845422 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:06.846094 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:06.846629 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:06.709928 16706 raft_consensus.cc:482] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:15:05.624533 16400 leader_election.cc:288] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:06.844823 13075 rpcz_store.cc:276] 0808 20:15:03.108843 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:03.109034 (+ 191us) service_pool.cc:227] Handling call
0808 20:15:03.222662 (+113628us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:05.265827 (+2043165us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:05.892730 (+626903us) spinlock_profiling.cc:243] Waited 2.04 s on lock 0x7b5400524ed4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:15:05.892952 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4697632512}
I20260808 20:15:06.710526 16599 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:06.848301 12250 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2097) took 4089 ms. Trace:
I20260808 20:15:06.851204 12250 rpcz_store.cc:276] 0808 20:15:02.759311 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.847601 (+4088290us) service_pool.cc:227] Handling call
0808 20:15:06.847858 (+ 257us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:06.848025 (+ 167us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:06.848048 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:06.848084 (+ 36us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:15:06.848117 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:06.848164 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:06.848256 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:06.852805 12250 raft_consensus.cc:1261] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 20:15:06.734032 16765 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:06.735308 16861 raft_consensus.cc:504] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:06.737566 16936 raft_consensus.cc:2781] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 20:15:06.743290 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:06.340384 16661 raft_consensus.cc:482] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:15:06.749061 16983 raft_consensus.cc:2781] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260808 20:15:06.352830 12043 rpcz_store.cc:269] 0808 20:15:02.540597 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.376750 (+2836153us) service_pool.cc:227] Handling call
0808 20:15:05.491968 (+115218us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:15:05.615370 12207 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.774s user 0.012s sys 0.068s
I20260808 20:15:06.749173 12039 raft_consensus.cc:2454] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
W20260808 20:15:06.856040 12039 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53018 (request call id 1614) took 2988 ms (client timeout 1963 ms). Trace:
W20260808 20:15:06.856405 12039 rpcz_store.cc:269] 0808 20:15:03.868061 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.474733 (+1606672us) service_pool.cc:227] Handling call
0808 20:15:06.855973 (+1381240us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:06.710646 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:06.746204 16817 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:06.758457 12144 rpcz_store.cc:276] 0808 20:15:02.900484 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.942284 (+ 41800us) service_pool.cc:227] Handling call
0808 20:15:02.953641 (+ 11357us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:04.680721 (+1727080us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:04.680753 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:04.680795 (+ 42us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:15:04.680824 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:05.726997 (+1046173us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b54000609d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:15:05.727046 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:05.727192 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3971436288}
I20260808 20:15:06.765025 16949 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:15:06.863528 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 115244us
I20260808 20:15:06.864370 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
W20260808 20:15:06.864377 12047 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 2148) took 1423 ms (client timeout 1869 ms). Trace:
W20260808 20:15:06.865382 12047 rpcz_store.cc:269] 0808 20:15:05.440948 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.710393 (+1269445us) service_pool.cc:227] Handling call
0808 20:15:06.864314 (+153921us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:06.864395 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:15:06.791495 12001 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.245s user 0.030s sys 0.100s
I20260808 20:15:06.867761 12144 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:06.785271 13076 rpcz_store.cc:276] 0808 20:15:04.912374 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.326573 (+414199us) service_pool.cc:227] Handling call
0808 20:15:06.599446 (+1272873us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54005b4854. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682e07c5 00007fa9682d8c1a 00007fa96d21c6d9 00007fa966906184 00007fa966906112 00007fa966906041 00007fa966905fbc 00007fa966905272 00007fa96670494c 00007fa966703d86
0808 20:15:06.599663 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2921054080}
I20260808 20:15:06.876926 12205 leader_election.cc:302] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:15:06.776942 16588 raft_consensus.cc:3037] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:15:06.767066 16788 raft_consensus.cc:3037] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:06.766446 13016 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:15:06.796970 12241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2096) took 4135 ms. Trace:
W20260808 20:15:06.887534 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 121159us
W20260808 20:15:05.816385 12251 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44384 (request call id 1953) took 2643 ms (client timeout 2295 ms). Trace:
I20260808 20:15:06.780735 12243 rpcz_store.cc:276] 0808 20:15:04.931889 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.779878 (+1847989us) service_pool.cc:227] Handling call
0808 20:15:06.780486 (+ 608us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:06.887557 12241 rpcz_store.cc:276] 0808 20:15:02.661336 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.794215 (+4132879us) service_pool.cc:227] Handling call
0808 20:15:06.794496 (+ 281us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:06.794680 (+ 184us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:06.794703 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:06.796686 (+ 1983us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:15:06.796733 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:06.796790 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:06.796913 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:06.775110 12254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2121) took 1966 ms. Trace:
I20260808 20:15:06.781407 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:06.602900 16649 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:15:06.781673 16991 raft_consensus.cc:3037] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:06.790165 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:06.888610 12254 rpcz_store.cc:276] 0808 20:15:04.808652 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.774100 (+1965448us) service_pool.cc:227] Handling call
0808 20:15:06.775063 (+ 963us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:06.841503 16901 raft_consensus.cc:504] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:15:05.935690 13079 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57330 (request call id 548) took 3411 ms (client timeout 1788 ms). Trace:
I20260808 20:15:06.889348 13016 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:15:06.844617 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:06.890457 12140 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 7.
I20260808 20:15:06.891413 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:06.891978 12140 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:15:06.790088 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:06.888545 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:06.892650 12152 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:06.893095 12243 raft_consensus.cc:2454] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6.
I20260808 20:15:06.729771 16892 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:15:06.843807 12142 rpcz_store.cc:269] 0808 20:15:02.274934 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.418549 (+143615us) service_pool.cc:227] Handling call
0808 20:15:06.646590 (+4228041us) spinlock_profiling.cc:243] Waited 4.07 ms on lock 0x7b540030c354. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:15:06.646718 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9362688}
W20260808 20:15:06.889580 13079 rpcz_store.cc:269] 0808 20:15:02.524252 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.524382 (+ 130us) service_pool.cc:227] Handling call
0808 20:15:05.935516 (+3411134us) spinlock_profiling.cc:243] Waited 2.11 s on lock 0x7b540044cad4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:15:05.935613 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4847573888}
I20260808 20:15:06.900774 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:15:06.847847 12248 rpcz_store.cc:269] 0808 20:15:02.855814 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.814640 (+2958826us) service_pool.cc:214] Skipping call since client already timed out
0808 20:15:05.814774 (+ 134us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 20:15:06.901369 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:15:06.842896 12247 rpcz_store.cc:269] 0808 20:15:04.820314 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.455415 (+635101us) service_pool.cc:227] Handling call
0808 20:15:06.842499 (+1387084us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:06.790807 12038 rpcz_store.cc:276] 0808 20:15:05.437347 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.503039 (+ 65692us) service_pool.cc:227] Handling call
0808 20:15:06.624683 (+1121644us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:06.790485 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:06.900447 12000 leader_election.cc:302] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:06.862334 16983 raft_consensus.cc:482] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:06.904512 12146 raft_consensus.cc:2379] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 9: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:15:06.904716 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:15:06.849321 12041 rpcz_store.cc:269] 0808 20:15:01.926362 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:02.736799 (+810437us) service_pool.cc:227] Handling call
0808 20:15:05.889567 (+3152768us) spinlock_profiling.cc:243] Waited 2.4 s on lock 0x7b540009f954. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:15:05.889709 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5513105280}
I20260808 20:15:06.854231 12045 raft_consensus.cc:2454] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 7.
W20260808 20:15:06.906963 12045 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 2149) took 1454 ms (client timeout 1865 ms). Trace:
W20260808 20:15:06.907316 12045 rpcz_store.cc:269] 0808 20:15:05.452293 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.743025 (+1290732us) service_pool.cc:227] Handling call
0808 20:15:06.906907 (+163882us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:06.853386 12250 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2098) took 4032 ms. Trace:
I20260808 20:15:06.907999 12250 rpcz_store.cc:276] 0808 20:15:02.821026 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.852268 (+4031242us) service_pool.cc:227] Handling call
0808 20:15:06.852534 (+ 266us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:06.853183 (+ 649us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:06.853342 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:06.909513 12250 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2104) took 2893 ms. Trace:
I20260808 20:15:06.909834 12250 rpcz_store.cc:276] 0808 20:15:04.016397 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.908723 (+2892326us) service_pool.cc:227] Handling call
0808 20:15:06.909037 (+ 314us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:06.909207 (+ 170us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:06.909232 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:06.909274 (+ 42us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:15:06.909305 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:06.909351 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:06.909444 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:06.853233 16706 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:06.913389 12243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1952) took 3889 ms. Trace:
I20260808 20:15:06.913611 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:06.914103 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:06.915000 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:06.915107 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:06.915836 12045 raft_consensus.cc:2379] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 7: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:15:06.916205 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:06.881381 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:15:06.881932 12207 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.040s user 0.050s sys 0.104s
I20260808 20:15:06.775297 16675 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:06.883917 12137 raft_consensus.cc:2454] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:15:06.855636 16599 raft_consensus.cc:3037] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:06.855284 16936 raft_consensus.cc:482] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:15:06.846817 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 2152) took 1340 ms. Trace:
W20260808 20:15:06.845271 12101 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.823s user 0.038s sys 0.042s
W20260808 20:15:06.766223 11896 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:57474 (request call id 334) took 26443 ms (client timeout 15000 ms). Trace:
W20260808 20:15:06.888108 12251 rpcz_store.cc:269] 0808 20:15:03.173528 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.816171 (+2642643us) service_pool.cc:214] Skipping call since client already timed out
0808 20:15:05.816312 (+ 141us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 20:15:06.902503 12048 raft_consensus.cc:2454] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 7.
I20260808 20:15:06.868398 16830 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 20:15:06.849066 12040 rpcz_store.cc:269] 0808 20:15:02.469676 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:04.999707 (+2530031us) service_pool.cc:227] Handling call
0808 20:15:05.679001 (+679294us) spinlock_profiling.cc:243] Waited 55.7 ms on lock 0x7b5400465ad4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:15:05.679155 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":128016640}
I20260808 20:15:06.864753 16661 raft_consensus.cc:504] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:06.869303 16949 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:06.916692 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:06.938624 12046 raft_consensus.cc:2379] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 9: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
W20260808 20:15:07.011076 12048 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 2151) took 1511 ms (client timeout 1864 ms). Trace:
I20260808 20:15:06.927302 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.009920 12045 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:06.943971 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:06.918699 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:06.949990 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:06.929409 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:06.921748 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.015058 12142 raft_consensus.cc:3037] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:06.969383 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
W20260808 20:15:07.010020 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 1.98819e+06us
I20260808 20:15:06.944489 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:06.950253 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:06.944773 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:06.949694 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:06.939893 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.016733 13077 raft_consensus.cc:3037] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:06.947458 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:06.953401 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:06.935472 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:07.011750 12036 rpcz_store.cc:276] 0808 20:15:05.506687 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.845865 (+1339178us) service_pool.cc:227] Handling call
0808 20:15:06.846763 (+ 898us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.018325 13080 raft_consensus.cc:2454] T f4918de0ffe94198a66ac1975ddbc693 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
W20260808 20:15:07.008579 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 217725us
W20260808 20:15:07.011525 12048 rpcz_store.cc:269] 0808 20:15:05.499887 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.782961 (+1283074us) service_pool.cc:227] Handling call
0808 20:15:07.011015 (+228054us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:15:07.012809 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 2.17114e+06us
I20260808 20:15:07.014343 12038 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 4.
I20260808 20:15:07.022210 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:07.022833 12001 leader_election.cc:302] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:15:06.916805 12140 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
W20260808 20:15:06.999653 12105 leader_election.cc:334] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901): Timed out: RequestConsensusVote RPC to 127.10.24.4:36901 timed out after 1.704s (SENT)
I20260808 20:15:07.023615 12043 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 2145) took 1846 ms. Trace:
I20260808 20:15:07.017185 12141 raft_consensus.cc:2454] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 9.
I20260808 20:15:07.014899 13075 raft_consensus.cc:3037] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:07.020149 13071 raft_consensus.cc:2454] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:15:07.019762 13079 raft_consensus.cc:2454] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:15:07.026232 12001 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:15:07.024693 12043 rpcz_store.cc:276] 0808 20:15:05.177622 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.855358 (+1677736us) service_pool.cc:227] Handling call
0808 20:15:07.023558 (+168200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.024456 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:07.015484 12144 raft_consensus.cc:2454] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 9.
I20260808 20:15:07.027273 12140 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Advancing to term 10
I20260808 20:15:07.029062 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:07.030550 12001 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:15:07.021533 12250 raft_consensus.cc:2454] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 6.
I20260808 20:15:07.031234 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:07.032192 12036 raft_consensus.cc:2379] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 9: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:15:07.020417 12207 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:07.031616 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.033493 12251 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 3.
I20260808 20:15:07.034092 12207 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:07.034508 12251 raft_consensus.cc:1226] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 7. Current term is 8. Ops: []
I20260808 20:15:07.026948 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:06.949416 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.032123 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:07.036106 12040 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:07.039855 12251 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2106) took 2533 ms. Trace:
I20260808 20:15:07.032264 12250 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2105) took 2642 ms. Trace:
I20260808 20:15:07.040858 12250 rpcz_store.cc:276] 0808 20:15:04.389508 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.031566 (+2642058us) service_pool.cc:227] Handling call
0808 20:15:07.031819 (+ 253us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.031988 (+ 169us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.032011 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.032046 (+ 35us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:15:07.032074 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.032123 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.032214 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.043459 12250 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2109) took 2501 ms. Trace:
I20260808 20:15:07.043843 12250 rpcz_store.cc:276] 0808 20:15:04.542344 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.041752 (+2499408us) service_pool.cc:227] Handling call
0808 20:15:07.042024 (+ 272us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.042244 (+ 220us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.042274 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.042312 (+ 38us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:15:07.042339 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.042387 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.043397 (+ 1010us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.046649 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:06.882583 12205 leader_election.cc:302] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
W20260808 20:15:07.047709 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 165190us
I20260808 20:15:07.042290 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:07.048950 12137 raft_consensus.cc:2454] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8.
I20260808 20:15:07.049819 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:07.040287 12251 rpcz_store.cc:276] 0808 20:15:04.506292 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.034084 (+2527792us) service_pool.cc:227] Handling call
0808 20:15:07.034346 (+ 262us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.039632 (+ 5286us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.039805 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.009792 12243 rpcz_store.cc:276] 0808 20:15:03.024388 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.905212 (+3880824us) service_pool.cc:227] Handling call
0808 20:15:06.912940 (+ 7728us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:06.913114 (+ 174us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:06.913139 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:06.913171 (+ 32us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:15:06.913202 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:06.913249 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:06.913337 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.055305 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.055419 12243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 642) took 2128 ms. Trace:
I20260808 20:15:07.055860 12243 rpcz_store.cc:276] 0808 20:15:04.927115 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.054728 (+2127613us) service_pool.cc:227] Handling call
0808 20:15:07.054980 (+ 252us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.055151 (+ 171us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.055173 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.055208 (+ 35us) raft_consensus.cc:1631] Marking committed up to 10
0808 20:15:07.055236 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.055280 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.055371 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.057588 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:15:07.034245 12046 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53028 (request call id 2139) took 4456 ms (client timeout 2802 ms). Trace:
I20260808 20:15:07.062039 12041 raft_consensus.cc:3037] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:07.062891 12249 raft_consensus.cc:2454] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
W20260808 20:15:07.061757 12046 rpcz_store.cc:269] 0808 20:15:02.578112 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:05.177032 (+2598920us) service_pool.cc:227] Handling call
0808 20:15:07.033996 (+1856964us) spinlock_profiling.cc:243] Waited 1.76 s on lock 0x7b5400007b54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:15:07.034177 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4040978176}
I20260808 20:15:06.918656 16859 raft_consensus.cc:2726] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 20:15:07.064662 12244 raft_consensus.cc:2379] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 5: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:15:06.900704 16938 raft_consensus.cc:2726] T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:15:06.881724 16762 raft_consensus.cc:504] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:06.903604 16857 leader_election.cc:288] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 9 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:07.068476 12141 raft_consensus.cc:2379] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 9: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:15:07.093004 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.069787 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:06.917320 16867 raft_consensus.cc:2726] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 20:15:07.070664 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.104326 13080 raft_consensus.cc:2454] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 7.
I20260808 20:15:06.997828 16901 leader_election.cc:288] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:15:07.071623 12101 connection.cc:232] RPC call timeout handler was delayed by -1.9477866950910538s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1743s for a response
W20260808 20:15:07.108953 12101 leader_election.cc:334] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Timed out: RequestConsensusVote RPC to 127.10.24.3:38391 timed out after 1.704s (SENT)
I20260808 20:15:07.109333 12101 leader_election.cc:302] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:07.110848 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:07.061795 16775 raft_consensus.cc:2726] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 20:15:07.069272 12046 raft_consensus.cc:1226] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260808 20:15:07.074049 12040 raft_consensus.cc:2454] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 9.
I20260808 20:15:07.118304 12040 raft_consensus.cc:1226] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 4. Ops: []
I20260808 20:15:07.118551 13015 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:15:07.075186 12243 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 643) took 2147 ms. Trace:
I20260808 20:15:07.076682 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e393e90875847809e1ef94179c5b4ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.118865 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 649) took 5491 ms. Trace:
I20260808 20:15:07.119724 13074 raft_consensus.cc:2379] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 5: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:15:07.077359 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:06.790259 16971 raft_consensus.cc:504] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:07.012666 16936 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:06.944877 16989 raft_consensus.cc:2781] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:15:07.066044 16903 raft_consensus.cc:2781] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:15:07.078128 12140 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P 78b9263889cd4f1582578a3aa4f644d9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 10.
I20260808 20:15:07.085685 13071 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 8.
I20260808 20:15:07.086465 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:07.087090 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:07.042694 16477 consensus_queue.cc:1046] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.075s
I20260808 20:15:07.089550 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237b38cfed904b8395749a75666444e9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.090287 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:06.918390 16983 raft_consensus.cc:3037] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Advancing to term 5
W20260808 20:15:07.092305 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 205787us
I20260808 20:15:07.094034 12043 raft_consensus.cc:2379] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 9: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:15:07.150385 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.030019 16670 raft_consensus.cc:2781] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:15:07.102284 12142 raft_consensus.cc:2454] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
I20260808 20:15:07.037966 16829 leader_election.cc:288] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:07.105473 12105 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:15:07.138916 12046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 648) took 5511 ms. Trace:
I20260808 20:15:07.152938 12046 rpcz_store.cc:276] 0808 20:15:01.627159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.067949 (+5440790us) service_pool.cc:227] Handling call
0808 20:15:07.069074 (+ 1125us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:15:07.138662 (+ 69588us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.138860 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.155465 12046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1590) took 5523 ms. Trace:
I20260808 20:15:07.155701 12046 rpcz_store.cc:276] 0808 20:15:01.632016 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.154925 (+5522909us) service_pool.cc:227] Handling call
0808 20:15:07.155142 (+ 217us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.155274 (+ 132us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.155290 (+ 16us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.155313 (+ 23us) raft_consensus.cc:1631] Marking committed up to 8
0808 20:15:07.155328 (+ 15us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.155357 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.155420 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.156684 12046 raft_consensus.cc:1226] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 8. Current term is 9. Ops: [7.7-8.8]
W20260808 20:15:07.105654 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 71468us
I20260808 20:15:07.038508 16992 raft_consensus.cc:2726] T c0f516c855714940a41eac002203e145 P b9d94f35ea15421aa1d721201c782d8a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 20:15:07.042590 16864 raft_consensus.cc:2781] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:15:07.113154 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.113873 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.114394 13014 leader_election.cc:302] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:15:07.069445 16982 raft_consensus.cc:2781] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260808 20:15:07.072202 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 52968.2us
I20260808 20:15:07.146898 16938 raft_consensus.cc:2781] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 20:15:07.145426 16989 raft_consensus.cc:482] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:07.182813 16989 raft_consensus.cc:3037] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
W20260808 20:15:07.088832 12244 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44378 (request call id 2111) took 2545 ms (client timeout 2902 ms). Trace:
W20260808 20:15:07.185072 12244 rpcz_store.cc:269] 0808 20:15:04.542954 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.707488 (+2164534us) service_pool.cc:227] Handling call
0808 20:15:07.088633 (+381145us) spinlock_profiling.cc:243] Waited 356 ms on lock 0x7b54000bc2d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:15:07.088772 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":819175680}
I20260808 20:15:07.186192 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:07.186403 12207 leader_election.cc:302] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:15:07.187638 12244 raft_consensus.cc:1226] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 5. Current term is 6. Ops: []
I20260808 20:15:07.188222 12244 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 645) took 2258 ms. Trace:
W20260808 20:15:07.132740 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 84391.7us
I20260808 20:15:07.189229 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102eea36004446e5a71aaccebb9ad859" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.189456 16775 raft_consensus.cc:2741] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260808 20:15:07.031278 16830 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:07.189785 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.190328 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.190958 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.191380 16830 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:07.149578 12247 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2103) took 3297 ms. Trace:
I20260808 20:15:07.152168 13078 raft_consensus.cc:2354] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6.
W20260808 20:15:07.159754 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 45443.8us
I20260808 20:15:07.193369 13014 leader_election.cc:302] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:15:07.193567 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.189106 16434 consensus_queue.cc:1035] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.005s
I20260808 20:15:07.194164 13078 raft_consensus.cc:2454] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:15:07.169709 16982 raft_consensus.cc:482] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:15:07.194708 16982 raft_consensus.cc:3037] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:15:07.195139 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
W20260808 20:15:07.115425 17000 log.cc:924] Time spent T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20260808 20:15:07.196146 12251 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2118) took 2651 ms. Trace:
I20260808 20:15:07.191996 12247 rpcz_store.cc:276] 0808 20:15:03.852412 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.902096 (+3049684us) service_pool.cc:227] Handling call
0808 20:15:06.939316 (+ 37220us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:06.939517 (+ 201us) raft_consensus.cc:1520] Early marking committed up to index 7
0808 20:15:06.969487 (+ 29970us) log.cc:842] Serialized 14 byte log entry
0808 20:15:07.149283 (+179796us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.149330 (+ 47us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:15:07.149368 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.149427 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.149526 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":29642,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":24126,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":74,"thread_start_us":179408,"threads_started":2,"wal-append.queue_time_us":72154}
I20260808 20:15:07.197156 16938 raft_consensus.cc:686] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:15:07.198316 12247 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 647) took 2264 ms. Trace:
I20260808 20:15:07.198062 16938 consensus_queue.cc:237] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:07.198673 12247 rpcz_store.cc:276] 0808 20:15:04.934396 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.197596 (+2263200us) service_pool.cc:227] Handling call
0808 20:15:07.197867 (+ 271us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.198036 (+ 169us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.198060 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.198097 (+ 37us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:15:07.198123 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.198172 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.198268 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.200249 12247 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 648) took 2261 ms. Trace:
I20260808 20:15:07.197072 12251 rpcz_store.cc:276] 0808 20:15:04.549030 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.051534 (+2502504us) service_pool.cc:227] Handling call
0808 20:15:07.051798 (+ 264us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.052005 (+ 207us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 20:15:07.071855 (+ 19850us) log.cc:842] Serialized 14 byte log entry
0808 20:15:07.147164 (+ 75309us) log.cc:842] Serialized 14 byte log entry
0808 20:15:07.184298 (+ 37134us) spinlock_profiling.cc:243] Waited 3.05 ms on lock 0x7b18001a96e0. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96e2518b1 00007fa96e25e661 00007fa9653294ca 00007fa96ea0af99 00007fa9651ed990 00007fa9653af864 00007fa9653af155 00007fa9682d23d0 00007fa9682e4576 00007fa9682f50d7 00007fa9682f5058
0808 20:15:07.184623 (+ 325us) log.cc:842] Serialized 14 byte log entry
0808 20:15:07.184859 (+ 236us) log.cc:842] Serialized 14 byte log entry
0808 20:15:07.195767 (+ 10908us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.195815 (+ 48us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:15:07.195858 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.195996 (+ 138us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.196094 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32500,"prepare.queue_time_us":26403,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":40797,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":3170,"spinlock_wait_cycles":11083520,"thread_start_us":74272,"threads_started":1}
I20260808 20:15:07.203572 12251 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 650) took 1884 ms. Trace:
I20260808 20:15:07.170363 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.205102 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.149036 16861 leader_election.cc:288] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:07.206004 16971 leader_election.cc:288] T 5e931b9df2c74729a3771d5f0734f559 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:07.207417 16762 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 9 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:07.146797 16903 raft_consensus.cc:482] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:07.209111 16903 raft_consensus.cc:3037] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:15:07.164597 16864 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:07.214917 16864 raft_consensus.cc:3037] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:15:07.221246 16675 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:07.194177 13014 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:15:07.190927 13074 raft_consensus.cc:2454] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:15:07.147797 16817 leader_election.cc:288] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:07.182379 12248 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2101) took 3351 ms. Trace:
I20260808 20:15:07.191612 13073 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 4.
I20260808 20:15:07.196218 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.224949 12251 rpcz_store.cc:276] 0808 20:15:05.319486 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.202898 (+1883412us) service_pool.cc:227] Handling call
0808 20:15:07.203143 (+ 245us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.203297 (+ 154us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.203317 (+ 20us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.203352 (+ 35us) raft_consensus.cc:1631] Marking committed up to 8
0808 20:15:07.203380 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.203425 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.203524 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.224431 12248 rpcz_store.cc:276] 0808 20:15:03.831195 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.901844 (+3070649us) service_pool.cc:227] Handling call
0808 20:15:06.933817 (+ 31973us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.145870 (+212053us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.145900 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.145938 (+ 38us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:15:07.145972 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.182156 (+ 36184us) spinlock_profiling.cc:243] Waited 212 ms on lock 0x7b5400165b54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:15:07.182214 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.182326 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":487247616}
I20260808 20:15:07.223155 12241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44384 (request call id 1947) took 4295 ms. Trace:
I20260808 20:15:07.227046 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.189368 12205 leader_election.cc:302] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:15:07.227809 12251 raft_consensus.cc:2379] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 7: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:15:07.222916 13014 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:15:07.198197 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.228554 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.230435 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.231359 12251 raft_consensus.cc:2379] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:15:07.226951 12241 rpcz_store.cc:276] 0808 20:15:02.927358 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.889600 (+3962242us) service_pool.cc:227] Handling call
0808 20:15:06.891932 (+ 2332us) raft_consensus.cc:1457] Updating replica for 3 ops
0808 20:15:06.892301 (+ 369us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:06.892328 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0808 20:15:06.901620 (+ 9292us) log.cc:842] Serialized 78 byte log entry
0808 20:15:07.146633 (+245013us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:15:07.146672 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.146734 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:15:07.222939 (+ 76205us) raft_consensus.cc:1671] finished
0808 20:15:07.222963 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.223091 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":95118,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":123,"thread_start_us":195724,"threads_started":1,"wal-append.queue_time_us":303528}
I20260808 20:15:07.232211 12207 leader_election.cc:302] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:15:07.151979 16670 raft_consensus.cc:482] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:07.232344 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.200534 12247 rpcz_store.cc:276] 0808 20:15:04.939286 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.199535 (+2260249us) service_pool.cc:227] Handling call
0808 20:15:07.199828 (+ 293us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.199980 (+ 152us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.200003 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.200037 (+ 34us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:15:07.200070 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.200117 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.200206 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.233556 13014 leader_election.cc:302] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:15:07.178774 16765 raft_consensus.cc:2781] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:15:07.234349 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.230412 16853 raft_consensus.cc:2781] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:15:07.241641 12249 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 644) took 2313 ms. Trace:
I20260808 20:15:07.239455 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:15:07.241127 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 51858.5us
I20260808 20:15:07.243448 12249 rpcz_store.cc:276] 0808 20:15:04.928437 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.063554 (+2135117us) service_pool.cc:227] Handling call
0808 20:15:07.063818 (+ 264us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.165517 (+101699us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.165543 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.165605 (+ 62us) raft_consensus.cc:1631] Marking committed up to 12
0808 20:15:07.165633 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.241406 (+ 75773us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b540007e754. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:15:07.241456 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.241576 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":233472256}
I20260808 20:15:07.244493 12205 leader_election.cc:302] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:15:07.235931 12241 raft_consensus.cc:2379] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:15:07.244972 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.245720 12249 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:15:07.245844 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.243821 12248 raft_consensus.cc:2454] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 5.
I20260808 20:15:07.246518 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:07.246752 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41c716cfc94652a2fc6957a1bd2a78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.249334 12205 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:15:07.188506 12244 rpcz_store.cc:276] 0808 20:15:04.929741 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.187147 (+2257406us) service_pool.cc:227] Handling call
0808 20:15:07.187458 (+ 311us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.188016 (+ 558us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.188177 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.251299 12205 leader_election.cc:302] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:15:07.252315 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.252764 12205 leader_election.cc:302] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:07.186334 16949 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:07.119333 12243 rpcz_store.cc:276] 0808 20:15:04.927804 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.056781 (+2128977us) service_pool.cc:227] Handling call
0808 20:15:07.057048 (+ 267us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.074864 (+ 17816us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.074892 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.074934 (+ 42us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:15:07.074963 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.075022 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.075135 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.252915 12251 raft_consensus.cc:2454] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 4.
I20260808 20:15:07.247570 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:06.969153 11889 catalog_manager.cc:5495] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 4 to 5, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 5 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } }
W20260808 20:15:06.969384 11965 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet cb8b79bb1c59483e93af813653a73a31 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e5bd9a4616ce4acfa20dc7a002984a20 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
W20260808 20:15:07.010511 11896 rpcz_store.cc:269] 0808 20:14:40.360029 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:14:40.365491 (+ 5462us) service_pool.cc:227] Handling call
0808 20:14:43.011812 (+2646321us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b5400002dd4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825faf0 00007fa9682e2f2a 00007fa96e234d6c 00007fa96e32ec3e 00007fa96c640fb4 00007fa96c640f42 00007fa96c640e71 00007fa96c640dec 00007fa96c6400a2 00007fa96670494c 00007fa966703d86
0808 20:15:06.766123 (+23754311us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 20:14:59.520724 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 20:14:59.532386 (+ 11662us) write_op.cc:432] Acquiring schema lock in shared mode
0808 20:14:59.532448 (+ 62us) write_op.cc:435] Acquired schema lock
0808 20:14:59.532465 (+ 17us) tablet.cc:660] Decoding operations
0808 20:14:59.533532 (+ 1067us) write_op.cc:620] Acquiring the partition lock for write op
0808 20:14:59.533623 (+ 91us) write_op.cc:641] Partition lock acquired for write op
0808 20:14:59.533648 (+ 25us) tablet.cc:683] Acquiring locks for 11 operations
0808 20:14:59.534635 (+ 987us) tablet.cc:699] Row locks acquired
0808 20:14:59.534675 (+ 40us) write_op.cc:260] PREPARE: finished
0808 20:14:59.534826 (+ 151us) write_op.cc:270] Start()
0808 20:14:59.534970 (+ 144us) write_op.cc:276] Timestamp: P: 1786220099534764 usec, L: 0
0808 20:14:59.534992 (+ 22us) op_driver.cc:331] REPLICATION: starting
0808 20:14:59.535548 (+ 556us) log.cc:842] Serialized 4388 byte log entry
0808 20:15:02.832336 (+3296788us) spinlock_profiling.cc:243] Waited 1.31 ms on lock 0x7b44000232cc. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa9651f6f11 00007fa9652c4a95 00007fa9652c54b2 00007fa967f6d511 00007fa967f68e1a 00007fa967f5cbee 00007fa967f5bf36 00007fa967f5bb58 00007fa967f65551 00007fa967f65509 00007fa967f65499
0808 20:15:04.751264 (+1918928us) op_driver.cc:444] REPLICATION: finished
0808 20:15:05.680721 (+929457us) write_op.cc:301] APPLY: starting
0808 20:15:05.685347 (+ 4626us) tablet.cc:1365] starting BulkCheckPresence
0808 20:15:05.726093 (+ 40746us) tablet.cc:1368] finished BulkCheckPresence
0808 20:15:05.726120 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 20:15:05.915582 (+189462us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 20:15:05.970028 (+ 54446us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0808 20:15:05.992329 (+ 22301us) write_op.cc:312] APPLY: finished
0808 20:15:06.004980 (+ 12651us) log.cc:842] Serialized 83 byte log entry
0808 20:15:06.390510 (+385530us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:15:06.390666 (+ 156us) write_op.cc:454] Released schema lock
0808 20:15:06.390713 (+ 47us) write_op.cc:341] FINISH: Updating metrics
0808 20:15:06.404139 (+ 13426us) write_op.cc:489] Releasing partition, row and schema locks
0808 20:15:06.404175 (+ 36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1039423,"reported_tablets":32,"spinlock_wait_cycles":2824849280,"child_traces":[["op",{"apply.queue_time_us":915265,"apply.run_cpu_time_us":10077,"apply.run_wall_time_us":734476,"num_ops":11,"prepare.queue_time_us":199309,"prepare.run_cpu_time_us":16222,"prepare.run_wall_time_us":303402,"raft.queue_time_us":1916273,"raft.run_cpu_time_us":167903,"raft.run_wall_time_us":1986163,"replication_time_us":5216085,"spinlock_wait_cycles":3019520,"thread_start_us":2894647,"threads_started":5,"wal-append.queue_time_us":949785,"wal-append.run_cpu_time_us":53748,"wal-append.run_wall_time_us":4146511}]]}
I20260808 20:15:07.125628 12041 raft_consensus.cc:2454] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 7.
I20260808 20:15:07.145216 12045 raft_consensus.cc:2454] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 9.
W20260808 20:15:07.138249 12101 connection.cc:232] RPC call timeout handler was delayed by -1.3366937430109829s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3155s for a response
I20260808 20:15:07.148180 12144 raft_consensus.cc:2454] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
W20260808 20:15:07.583302 12101 connection.cc:232] RPC call timeout handler was delayed by -1.2736866951454431s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2492s for a response
I20260808 20:15:07.354780 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.584980 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.550449 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 2124) took 5953 ms. Trace:
W20260808 20:15:07.488866 11994 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.24.3:38391 timed out after 1.747s (SENT) [suppressed 62 similar messages]
W20260808 20:15:07.521867 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 363074us
I20260808 20:15:07.586840 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:15:07.586326 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 333536us
I20260808 20:15:07.590261 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:15:07.588742 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 100096us
I20260808 20:15:07.591285 12243 raft_consensus.cc:2454] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:15:07.592110 12205 leader_election.cc:302] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:15:07.586503 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.504855 12038 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 7.
I20260808 20:15:07.586687 12036 rpcz_store.cc:276] 0808 20:15:01.596928 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.032956 (+5436028us) service_pool.cc:227] Handling call
0808 20:15:07.033307 (+ 351us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:15:07.070394 (+ 37087us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.070425 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:15:07.105975 (+ 35550us) log.cc:842] Serialized 26 byte log entry
0808 20:15:07.527634 (+421659us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:15:07.528449 (+ 815us) log.cc:842] Serialized 14 byte log entry
0808 20:15:07.528811 (+ 362us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.528941 (+ 130us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:15:07.550033 (+ 21092us) raft_consensus.cc:1671] finished
0808 20:15:07.550053 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.550281 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5723,"prepare.queue_time_us":36134,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":5853,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":15033,"thread_start_us":15578,"threads_started":1,"wal-append.queue_time_us":406268}
I20260808 20:15:07.594484 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.595472 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:07.512610 12043 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 651) took 5884 ms. Trace:
I20260808 20:15:07.513471 12048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 650) took 5885 ms. Trace:
W20260808 20:15:07.597304 11965 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 652132us
I20260808 20:15:07.596302 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:07.267002 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.320060 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.606340 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:07.606936 12248 raft_consensus.cc:2454] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:15:07.607087 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:15:07.607782 12101 leader_election.cc:334] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Timed out: RequestConsensusVote RPC to 127.10.24.3:38391 timed out after 1.743s (SENT)
I20260808 20:15:07.600594 12040 rpcz_store.cc:276] 0808 20:15:01.627453 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.117870 (+5490417us) service_pool.cc:227] Handling call
0808 20:15:07.118145 (+ 275us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.118655 (+ 510us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.118817 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:15:07.619024 12101 leader_election.cc:334] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Timed out: RequestConsensusVote RPC to 127.10.24.3:38391 timed out after 2.054s (SENT)
I20260808 20:15:07.602895 12241 raft_consensus.cc:2454] T 0d91cb12be034dcc9ca3038df3cc85cf P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
I20260808 20:15:07.320936 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.608707 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:07.608721 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.599586 12205 leader_election.cc:302] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:15:07.567560 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 2122) took 6413 ms. Trace:
I20260808 20:15:07.597034 12043 rpcz_store.cc:276] 0808 20:15:01.627930 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.151056 (+5523126us) service_pool.cc:227] Handling call
0808 20:15:07.511664 (+360608us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.512116 (+ 452us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.512156 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.512233 (+ 77us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:15:07.512309 (+ 76us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.512355 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.512452 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.607447 12249 raft_consensus.cc:2379] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 7: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:15:07.611748 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.597363 12048 rpcz_store.cc:276] 0808 20:15:01.627697 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.149752 (+5522055us) service_pool.cc:227] Handling call
0808 20:15:07.512664 (+362912us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.513075 (+ 411us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.513099 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.513152 (+ 53us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:15:07.513177 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.513227 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.513319 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.613605 13015 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
W20260808 20:15:07.403904 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 178123us
I20260808 20:15:07.605528 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 655) took 5887 ms. Trace:
I20260808 20:15:07.511924 12046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1596) took 5871 ms. Trace:
I20260808 20:15:07.621557 12045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 654) took 5908 ms. Trace:
I20260808 20:15:07.592715 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.627450 12036 rpcz_store.cc:276] 0808 20:15:01.718298 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.604409 (+5886111us) service_pool.cc:227] Handling call
0808 20:15:07.604785 (+ 376us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.605192 (+ 407us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.605227 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.605267 (+ 40us) raft_consensus.cc:1631] Marking committed up to 10
0808 20:15:07.605303 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.605354 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.605451 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.627666 12046 rpcz_store.cc:276] 0808 20:15:01.640028 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.156304 (+5516276us) service_pool.cc:227] Handling call
0808 20:15:07.156533 (+ 229us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 20:15:07.511135 (+354602us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.511689 (+ 554us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.627751 12045 rpcz_store.cc:276] 0808 20:15:01.713410 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.603714 (+5890304us) service_pool.cc:227] Handling call
0808 20:15:07.604101 (+ 387us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.621093 (+ 16992us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.621118 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.621174 (+ 56us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:15:07.621206 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.621259 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.621367 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:15:07.483422 11943 debug-util.cc:398] Leaking SignalData structure 0x7b08001f5a80 after lost signal to thread 10336
W20260808 20:15:07.685359 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 86030.8us
W20260808 20:15:07.685235 11943 debug-util.cc:398] Leaking SignalData structure 0x7b08001f5e60 after lost signal to thread 10340
I20260808 20:15:07.234923 16670 raft_consensus.cc:3037] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:07.684216 12038 raft_consensus.cc:2454] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 7.
I20260808 20:15:07.235827 16762 raft_consensus.cc:2781] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:15:07.592666 11889 catalog_manager.cc:5495] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 4 to 7, leader changed from e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 7 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:15:07.694008 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 2137) took 5181 ms. Trace:
I20260808 20:15:07.203886 16788 raft_consensus.cc:2726] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:15:07.694780 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7d7a85e3964f3f9d5f2dc39e23340b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.695528 12247 raft_consensus.cc:2454] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 3.
I20260808 20:15:07.250463 16971 raft_consensus.cc:2781] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:15:07.164860 16881 raft_consensus.cc:3032] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 LEADER]: Stepping down as leader of term 4
W20260808 20:15:07.694263 11943 debug-util.cc:398] Leaking SignalData structure 0x7b08003cf820 after lost signal to thread 11946
I20260808 20:15:07.696326 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 2130) took 5557 ms. Trace:
I20260808 20:15:07.238183 16765 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:15:07.712989 12038 rpcz_store.cc:276] 0808 20:15:02.512165 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.692602 (+5180437us) service_pool.cc:227] Handling call
0808 20:15:07.693082 (+ 480us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.693554 (+ 472us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.693589 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.693678 (+ 89us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:15:07.693745 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.693824 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.693940 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.716393 12036 rpcz_store.cc:276] 0808 20:15:02.139070 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.634437 (+5495367us) service_pool.cc:227] Handling call
0808 20:15:07.695530 (+ 61093us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.695861 (+ 331us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.695900 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.695978 (+ 78us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:15:07.696066 (+ 88us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.696144 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.696250 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.236620 16853 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:15:07.126497 15674 raft_consensus.cc:2726] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:15:07.261206 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.443241 16864 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:15:07.730058 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 116833us
I20260808 20:15:07.730598 16675 raft_consensus.cc:2781] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:15:07.731132 16675 raft_consensus.cc:482] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:07.319667 16706 raft_consensus.cc:504] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true } }
I20260808 20:15:07.280814 16599 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:07.733078 16881 raft_consensus.cc:729] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:15:07.733778 16706 leader_election.cc:288] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 9 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:07.250705 16942 raft_consensus.cc:2781] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:15:07.388846 16983 raft_consensus.cc:504] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:07.734678 16881 consensus_queue.cc:260] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:07.273048 16588 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:07.735960 17010 raft_consensus.cc:2781] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:15:07.736419 16881 raft_consensus.cc:3037] T a16032fd7a244ded8d9660c183645279 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:15:07.736536 17010 raft_consensus.cc:482] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:15:07.416309 16859 raft_consensus.cc:2781] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:15:07.737038 17010 raft_consensus.cc:3037] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:15:07.737874 16762 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:07.738134 16588 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:07.738595 16762 raft_consensus.cc:3037] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:07.739607 12254 raft_consensus.cc:2379] T eec183c460ad4f1b85ad1407c53f2eab P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 9: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:15:07.740571 12147 raft_consensus.cc:2379] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:15:07.256574 16661 leader_election.cc:288] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:07.254602 13080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:07.253554 16829 raft_consensus.cc:2781] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:15:07.440183 16982 raft_consensus.cc:504] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:07.741813 16829 raft_consensus.cc:482] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:07.742450 16829 raft_consensus.cc:3037] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:07.742844 12045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 2134) took 5283 ms. Trace:
I20260808 20:15:07.253350 16434 raft_consensus.cc:2781] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:15:07.293396 13077 raft_consensus.cc:2454] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
W20260808 20:15:07.728343 11943 debug-util.cc:398] Leaking SignalData structure 0x7b0800351d00 after lost signal to thread 11966
W20260808 20:15:07.717764 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 114545us
I20260808 20:15:07.720672 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.743278 12045 rpcz_store.cc:276] 0808 20:15:02.459096 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.698114 (+5239018us) service_pool.cc:227] Handling call
0808 20:15:07.742162 (+ 44048us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.742546 (+ 384us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.742572 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.742611 (+ 39us) raft_consensus.cc:1631] Marking committed up to 5
0808 20:15:07.742636 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.742683 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.742782 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.254161 17009 raft_consensus.cc:2781] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:15:07.721772 12243 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
I20260808 20:15:07.696359 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.315331 13075 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 9.
I20260808 20:15:07.625438 12039 rpcz_store.cc:276] 0808 20:15:01.153722 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.864221 (+5710499us) service_pool.cc:227] Handling call
0808 20:15:06.887715 (+ 23494us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:15:06.904056 (+ 16341us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:06.904086 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:15:06.911042 (+ 6956us) log.cc:842] Serialized 26 byte log entry
0808 20:15:06.988497 (+ 77455us) spinlock_profiling.cc:243] Waited 7.11 ms on lock 0x7b180023cc80. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96e2518b1 00007fa96e25e661 00007fa965394345 00007fa96e1cba39 00007fa9653b18bb 00007fa9653afcbf 00007fa9653b1e22 00007fa967f5b5bb 00007fa967f61063 00007fa967f61411 00007fa9682ae5cf
0808 20:15:07.084487 (+ 95990us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:15:07.566413 (+481926us) log.cc:842] Serialized 14 byte log entry
0808 20:15:07.566837 (+ 424us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.566971 (+ 134us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:15:07.567037 (+ 66us) raft_consensus.cc:1671] finished
0808 20:15:07.567153 (+ 116us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.567353 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15982,"prepare.queue_time_us":95595,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":16370,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":17621,"spinlock_wait_cycles":16362368,"thread_start_us":575824,"threads_started":2,"wal-append.queue_time_us":280028}
I20260808 20:15:07.734700 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:07.253692 17008 raft_consensus.cc:2726] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 20:15:07.747632 13072 raft_consensus.cc:2454] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
W20260808 20:15:07.422266 16508 consensus_meta.cc:287] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.553s user 0.008s sys 0.000s
I20260808 20:15:07.754797 16992 raft_consensus.cc:2781] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:15:07.755146 13078 raft_consensus.cc:1226] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 LEADER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260808 20:15:07.756033 17025 raft_consensus.cc:2781] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:15:07.756567 17025 raft_consensus.cc:482] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:15:07.756670 16765 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:15:07.287465 16936 raft_consensus.cc:504] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:07.418812 16903 raft_consensus.cc:504] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:15:07.748286 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 112492us
I20260808 20:15:07.752242 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5dc2ab0712f4054883178f4b4c50928" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.731652 16675 raft_consensus.cc:3037] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
W20260808 20:15:07.721978 12101 leader_election.cc:334] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391): Timed out: RequestConsensusVote RPC to 127.10.24.3:38391 timed out after 2.492s (SENT)
I20260808 20:15:07.741163 12205 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:07.769481 12146 raft_consensus.cc:2354] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 8. Current term is 9.
I20260808 20:15:07.735159 16599 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:07.767760 16857 raft_consensus.cc:2726] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:15:07.744989 16971 raft_consensus.cc:482] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
W20260808 20:15:07.816109 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 127865us
I20260808 20:15:07.818063 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1599) took 5851 ms. Trace:
W20260808 20:15:07.819293 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 86842.6us
I20260808 20:15:07.692694 16779 consensus_queue.cc:1046] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.077s
I20260808 20:15:07.773509 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 659) took 5110 ms. Trace:
I20260808 20:15:07.765877 17025 raft_consensus.cc:3037] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:15:07.760814 16936 leader_election.cc:288] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:07.754879 16508 raft_consensus.cc:504] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:07.763464 16903 leader_election.cc:288] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:07.819375 12040 rpcz_store.cc:276] 0808 20:15:01.919824 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.617926 (+5698102us) service_pool.cc:227] Handling call
0808 20:15:07.618471 (+ 545us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.618907 (+ 436us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 20:15:07.620249 (+ 1342us) log.cc:842] Serialized 14 byte log entry
0808 20:15:07.771311 (+151062us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.771358 (+ 47us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:15:07.771398 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.771457 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.771570 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56336,"prepare.queue_time_us":774,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":63524,"thread_start_us":70152,"threads_started":1}
I20260808 20:15:07.829572 17009 raft_consensus.cc:482] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:07.830468 12039 raft_consensus.cc:1226] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 6. Current term is 7. Ops: []
I20260808 20:15:07.830794 16983 leader_election.cc:288] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:07.830341 11889 catalog_manager.cc:5495] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 4 to 6, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4). New cstate: current_term: 6 leader_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:15:07.866631 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 2143) took 5039 ms. Trace:
I20260808 20:15:07.867009 12039 rpcz_store.cc:276] 0808 20:15:02.827102 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.772479 (+4945377us) service_pool.cc:227] Handling call
0808 20:15:07.830312 (+ 57833us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.866370 (+ 36058us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.866568 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.831898 12046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 2136) took 5320 ms. Trace:
I20260808 20:15:07.388552 16989 raft_consensus.cc:504] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:07.833307 12250 raft_consensus.cc:1226] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 4. Current term is 5. Ops: []
I20260808 20:15:07.834466 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:07.835109 16942 raft_consensus.cc:686] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:15:07.835276 16853 raft_consensus.cc:3037] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:15:07.822816 12038 rpcz_store.cc:276] 0808 20:15:02.657891 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.724308 (+5066417us) service_pool.cc:227] Handling call
0808 20:15:07.767665 (+ 43357us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.767964 (+ 299us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.767990 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.768033 (+ 43us) raft_consensus.cc:1631] Marking committed up to 7
0808 20:15:07.768070 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.768127 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.768231 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.839282 12001 leader_election.cc:302] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:15:07.831388 13074 raft_consensus.cc:1226] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7. Ops: []
I20260808 20:15:07.897634 12147 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1853) took 1110 ms. Trace:
I20260808 20:15:07.902047 12147 rpcz_store.cc:276] 0808 20:15:06.786848 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.786988 (+ 140us) service_pool.cc:227] Handling call
0808 20:15:07.897386 (+1110398us) spinlock_profiling.cc:243] Waited 850 ms on lock 0x7b54004259d4. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:15:07.897550 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1954456448}
I20260808 20:15:07.763669 16992 raft_consensus.cc:482] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:07.903859 16992 raft_consensus.cc:3037] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:15:07.905090 16989 leader_election.cc:288] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:07.878818 17009 raft_consensus.cc:3037] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
W20260808 20:15:07.841976 11943 debug-util.cc:398] Leaking SignalData structure 0x7b08001ef860 after lost signal to thread 12083
I20260808 20:15:07.823277 16971 raft_consensus.cc:3037] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
W20260808 20:15:07.914513 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 75560.2us
I20260808 20:15:07.915258 17010 raft_consensus.cc:504] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:07.916530 12001 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:15:07.844578 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:07.919673 13075 raft_consensus.cc:3037] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:15:07.919938 16867 raft_consensus.cc:2781] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:15:07.920385 16867 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:15:07.920683 16867 raft_consensus.cc:3037] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:15:07.846930 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940aa50483d844deb4031f312b4a7355" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
W20260808 20:15:07.833321 12152 consensus_meta.cc:287] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.940s user 0.010s sys 0.011s
I20260808 20:15:07.921243 12152 raft_consensus.cc:2454] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 9.
I20260808 20:15:07.884866 16670 raft_consensus.cc:504] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:07.921885 12152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1854) took 1134 ms. Trace:
I20260808 20:15:07.850332 12043 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 657) took 5885 ms. Trace:
I20260808 20:15:07.921984 16942 consensus_queue.cc:237] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:07.922312 12043 rpcz_store.cc:276] 0808 20:15:01.964602 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.632081 (+5667479us) service_pool.cc:227] Handling call
0808 20:15:07.723543 (+ 91462us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.825970 (+102427us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.825996 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.850042 (+ 24046us) raft_consensus.cc:1631] Marking committed up to 8
0808 20:15:07.850091 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.850159 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.850272 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.850620 16508 leader_election.cc:288] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:07.851096 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:07.890156 12102 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:15:07.924916 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e4bf0005d9e43c1a4adce7588e7653f" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:07.926275 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.926661 12038 raft_consensus.cc:1226] T 2cd3d6998af44e69a7182125e02f16e2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7. Ops: []
I20260808 20:15:07.927314 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1610) took 4985 ms. Trace:
I20260808 20:15:07.927459 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.851432 12047 raft_consensus.cc:1226] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 4. Current term is 5. Ops: []
I20260808 20:15:07.931185 12102 leader_election.cc:302] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:15:07.932056 12047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 647) took 6621 ms. Trace:
W20260808 20:15:07.854068 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 134189us
I20260808 20:15:07.854564 16859 raft_consensus.cc:482] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:07.932479 12047 rpcz_store.cc:276] 0808 20:15:01.310831 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.866056 (+5555225us) service_pool.cc:227] Handling call
0808 20:15:06.866349 (+ 293us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.931089 (+1064740us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.931749 (+ 660us) spinlock_profiling.cc:243] Waited 985 ms on lock 0x7b5400012554. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:15:07.931946 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2264582912}
I20260808 20:15:07.835050 16434 raft_consensus.cc:482] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:07.933733 16434 raft_consensus.cc:3037] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:15:07.934871 12047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 2150) took 2460 ms. Trace:
I20260808 20:15:07.935329 12047 rpcz_store.cc:276] 0808 20:15:05.474855 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.933732 (+2458877us) service_pool.cc:227] Handling call
0808 20:15:07.934190 (+ 458us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.934511 (+ 321us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.934540 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.934581 (+ 41us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:15:07.934623 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.934689 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.934811 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.404084 16991 raft_consensus.cc:504] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:07.843729 16864 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:07.843693 16706 raft_consensus.cc:2781] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:15:07.850745 12248 raft_consensus.cc:3037] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Advancing to term 10
I20260808 20:15:07.864907 15674 consensus_queue.cc:1035] T 3051f9a615344b5c80595ef0ace350c8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260808 20:15:07.868685 16982 leader_election.cc:288] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:07.869668 12045 raft_consensus.cc:1226] T b0f8a2331e88407c9e3da24439d202b2 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 8. Current term is 9. Ops: []
I20260808 20:15:07.869381 11889 catalog_manager.cc:5495] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 3 to 4, leader changed from e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 4 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 20:15:07.871887 16892 consensus_queue.cc:1046] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 20:15:07.873050 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.874008 17033 raft_consensus.cc:3032] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 20:15:07.942250 16706 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:07.868628 13076 raft_consensus.cc:2379] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 9: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:15:07.868162 12046 rpcz_store.cc:276] 0808 20:15:02.511797 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.698095 (+5186298us) service_pool.cc:227] Handling call
0808 20:15:07.745542 (+ 47447us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.831515 (+ 85973us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.831547 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.831604 (+ 57us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:15:07.831637 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.831710 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.831838 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.879735 12036 raft_consensus.cc:1226] T 102eea36004446e5a71aaccebb9ad859 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 8. Current term is 9. Ops: []
I20260808 20:15:07.949865 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 660) took 5244 ms. Trace:
W20260808 20:15:07.880543 12105 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.004s sys 0.032s
I20260808 20:15:07.880828 12101 leader_election.cc:302] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:15:07.900745 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 68245.1us
I20260808 20:15:07.891742 13015 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
W20260808 20:15:07.911597 11943 debug-util.cc:398] Leaking SignalData structure 0x7b080016dd00 after lost signal to thread 12185
I20260808 20:15:07.917196 17027 raft_consensus.cc:2781] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 20:15:07.844394 13077 raft_consensus.cc:3037] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 LEARNER]: Advancing to term 9
I20260808 20:15:07.918468 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:07.918690 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:07.929211 12039 raft_consensus.cc:1261] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Refusing update from remote peer 78b9263889cd4f1582578a3aa4f644d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260808 20:15:07.929729 16588 raft_consensus.cc:2781] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:15:07.930492 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.936349 12038 rpcz_store.cc:276] 0808 20:15:02.941502 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.926109 (+4984607us) service_pool.cc:227] Handling call
0808 20:15:07.926435 (+ 326us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.927061 (+ 626us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.927255 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.932469 12102 leader_election.cc:302] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters:
I20260808 20:15:07.950184 12036 rpcz_store.cc:276] 0808 20:15:02.705648 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.722394 (+5016746us) service_pool.cc:227] Handling call
0808 20:15:07.825253 (+102859us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.949644 (+124391us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.949813 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.937973 12152 rpcz_store.cc:276] 0808 20:15:06.787121 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.787221 (+ 100us) service_pool.cc:227] Handling call
0808 20:15:07.921810 (+1134589us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.951097 12045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53018 (request call id 1606) took 5191 ms. Trace:
I20260808 20:15:07.933300 16859 raft_consensus.cc:3037] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:07.961318 12046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 668) took 1069 ms. Trace:
I20260808 20:15:07.947875 17033 raft_consensus.cc:729] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:15:07.963385 17033 consensus_queue.cc:260] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:07.961351 16477 raft_consensus.cc:2781] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 20:15:07.957999 16983 consensus_queue.cc:1046] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
W20260808 20:15:07.958561 13015 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 66969.4us
I20260808 20:15:07.966672 16983 raft_consensus.cc:3032] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Stepping down as leader of term 6
I20260808 20:15:07.967123 16983 raft_consensus.cc:729] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:15:07.874790 16675 raft_consensus.cc:504] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:07.944478 17010 leader_election.cc:288] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:07.932050 16936 raft_consensus.cc:2781] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 20:15:07.969282 17010 consensus_queue.cc:1046] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 20:15:07.969947 16936 raft_consensus.cc:686] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:15:07.970472 17010 raft_consensus.cc:3032] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Stepping down as leader of term 6
I20260808 20:15:07.950503 12243 raft_consensus.cc:2379] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:15:07.971894 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e256c29dec964f49856cf204f950793b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.973663 12241 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 7.
I20260808 20:15:07.968283 16983 consensus_queue.cc:260] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:07.974634 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.975854 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0d98ab3b4945d8b845711f4ddd7153" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.977902 11994 leader_election.cc:302] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:07.978638 16762 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:07.952178 12045 rpcz_store.cc:276] 0808 20:15:02.760187 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.766098 (+5005911us) service_pool.cc:227] Handling call
0808 20:15:07.869399 (+103301us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.949099 (+ 79700us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.951045 (+ 1946us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.984623 12045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:53014 (request call id 1417) took 6859 ms. Trace:
I20260808 20:15:07.952809 16892 consensus_queue.cc:1225] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer b9d94f35ea15421aa1d721201c782d8a log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20260808 20:15:07.953437 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 72762us
I20260808 20:15:07.984948 12045 rpcz_store.cc:276] 0808 20:15:01.125649 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.983104 (+6857455us) service_pool.cc:227] Handling call
0808 20:15:07.984545 (+ 1441us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.950873 16706 raft_consensus.cc:3037] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:07.946557 12047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 2154) took 1060 ms. Trace:
I20260808 20:15:07.989076 12047 rpcz_store.cc:276] 0808 20:15:06.885267 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.939483 (+1054216us) service_pool.cc:227] Handling call
0808 20:15:07.941116 (+ 1633us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.945270 (+ 4154us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.945300 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.945338 (+ 38us) raft_consensus.cc:1631] Marking committed up to 2
0808 20:15:07.945363 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.945414 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.945531 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.956797 16903 consensus_queue.cc:1046] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 20:15:07.957196 17027 raft_consensus.cc:686] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:15:07.989809 17025 raft_consensus.cc:504] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:07.955626 16670 leader_election.cc:288] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:07.990715 13014 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:15:07.952749 12105 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.794s user 0.005s sys 0.032s
I20260808 20:15:07.991088 13079 raft_consensus.cc:2379] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 8: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:15:07.957952 16588 raft_consensus.cc:482] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:15:07.928793 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.985997 12101 leader_election.cc:302] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:15:07.961700 12046 rpcz_store.cc:276] 0808 20:15:06.891529 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.960287 (+1068758us) service_pool.cc:227] Handling call
0808 20:15:07.960741 (+ 454us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.961016 (+ 275us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.961038 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.961075 (+ 37us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:15:07.961100 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.961149 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.961249 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.973027 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.959949 16991 leader_election.cc:288] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:07.999192 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8a2331e88407c9e3da24439d202b2" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:07.958567 12039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 2147) took 2518 ms. Trace:
I20260808 20:15:07.999790 12039 rpcz_store.cc:276] 0808 20:15:05.439778 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.928488 (+2488710us) service_pool.cc:227] Handling call
0808 20:15:07.928853 (+ 365us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.951400 (+ 22547us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.958509 (+ 7109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.999153 13074 raft_consensus.cc:2454] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:15:07.970898 17010 raft_consensus.cc:729] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:15:07.979058 16992 raft_consensus.cc:504] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:07.979499 11994 leader_election.cc:302] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:07.961768 12040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 2144) took 5134 ms. Trace:
I20260808 20:15:08.000487 13077 raft_consensus.cc:1261] T 21889249378248049ef043f0588cebc9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 LEARNER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260808 20:15:07.953480 11889 catalog_manager.cc:5495] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 5 to 6, leader changed from 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 6 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: HEALTHY } } }
I20260808 20:15:08.003240 12040 rpcz_store.cc:276] 0808 20:15:02.827432 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.866912 (+5039480us) service_pool.cc:227] Handling call
0808 20:15:07.947217 (+ 80305us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.961470 (+ 14253us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.961496 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:07.961534 (+ 38us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:15:07.961568 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.961620 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:07.961718 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:07.981451 12036 raft_consensus.cc:1226] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 6. Current term is 7. Ops: []
I20260808 20:15:07.897060 16829 raft_consensus.cc:504] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:15:07.982333 11943 debug-util.cc:398] Leaking SignalData structure 0x7b08003c6c80 after lost signal to thread 12287
I20260808 20:15:07.980046 16982 raft_consensus.cc:2781] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 20:15:07.954110 17017 raft_consensus.cc:2726] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:15:07.958822 16400 raft_consensus.cc:2781] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:15:07.960744 12245 raft_consensus.cc:1226] T aacf7252c47a48cdb408ea9e49270f80 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 6. Current term is 7. Ops: []
I20260808 20:15:08.014624 16892 raft_consensus.cc:2726] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 20:15:08.017268 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:08.018124 12244 raft_consensus.cc:2379] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 8: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:15:08.018898 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:08.019583 12244 raft_consensus.cc:2379] T ac9505fd371049b7b3ca47aad176ad64 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 7: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:15:08.020308 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:08.008380 16982 raft_consensus.cc:482] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:08.021234 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f516c855714940a41eac002203e145" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:08.021863 12038 raft_consensus.cc:1226] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 6. Current term is 7. Ops: []
I20260808 20:15:08.022311 16829 leader_election.cc:288] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:07.999459 16588 raft_consensus.cc:3037] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:15:07.928793 16765 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.026324 16765 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:07.992723 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:08.027889 16706 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.029202 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:08.030408 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:08.031840 12241 raft_consensus.cc:1226] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 5. Current term is 6. Ops: []
W20260808 20:15:08.001997 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 896524us
I20260808 20:15:08.035271 16942 raft_consensus.cc:2726] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 20:15:08.036211 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:08.037746 13080 raft_consensus.cc:1226] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 4. Current term is 5. Ops: []
I20260808 20:15:08.038549 12152 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64"
dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:49548
I20260808 20:15:08.039310 12152 raft_consensus.cc:595] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:15:08.039714 12152 raft_consensus.cc:603] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 20:15:08.040103 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:08.021248 16982 raft_consensus.cc:3037] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:15:08.041347 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:07.982530 16675 leader_election.cc:288] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:07.948475 17009 raft_consensus.cc:504] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:08.043972 17009 leader_election.cc:288] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:08.022498 12038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 2157) took 1119 ms. Trace:
I20260808 20:15:08.046006 16762 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:08.046389 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:08.022456 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:08.047127 13072 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:07.990466 16903 raft_consensus.cc:3032] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 8 LEADER]: Stepping down as leader of term 8
I20260808 20:15:08.048460 16903 raft_consensus.cc:729] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:15:08.049057 12249 raft_consensus.cc:2454] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
I20260808 20:15:07.964685 17033 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:08.054106 16859 raft_consensus.cc:504] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:15:08.074265 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 94888us
I20260808 20:15:08.011994 16400 raft_consensus.cc:482] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:15:08.075600 16400 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:15:08.075742 16859 leader_election.cc:288] T f73cf4b21a754e75b2af936d7402def2 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 7 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:08.076113 11994 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:08.076920 16864 raft_consensus.cc:2781] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:15:08.077298 16864 raft_consensus.cc:482] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:08.077634 16864 raft_consensus.cc:3037] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:08.046607 17010 consensus_queue.cc:260] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.080407 17010 raft_consensus.cc:3037] T 4e4bf0005d9e43c1a4adce7588e7653f P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:07.986815 16971 raft_consensus.cc:504] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:15:08.018126 11943 debug-util.cc:398] Leaking SignalData structure 0x7b0800133740 after lost signal to thread 13118
I20260808 20:15:08.045424 12038 rpcz_store.cc:276] 0808 20:15:06.903279 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:08.007518 (+1104239us) service_pool.cc:227] Handling call
0808 20:15:08.021659 (+ 14141us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:08.022262 (+ 603us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:08.022434 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:15:08.035554 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 90922.1us
I20260808 20:15:08.081962 11994 leader_election.cc:302] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:08.088972 11994 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:08.089087 16508 raft_consensus.cc:2781] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:15:08.089541 16508 raft_consensus.cc:482] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:08.089946 16508 raft_consensus.cc:3037] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:15:08.091671 12147 raft_consensus.cc:1261] T 9d6f3bbe5e92475487be203c7c89b73d P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260808 20:15:08.065706 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:08.047283 12243 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 3.
I20260808 20:15:08.050877 12244 raft_consensus.cc:2379] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 5: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:15:08.055064 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:08.057111 16670 raft_consensus.cc:2726] T 2a41c716cfc94652a2fc6957a1bd2a78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 20:15:08.057206 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:15:08.057740 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 71814.7us
I20260808 20:15:08.058022 12146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53176 (request call id 1868) took 1205 ms. Trace:
I20260808 20:15:08.058846 12250 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2115) took 3514 ms. Trace:
I20260808 20:15:08.059561 12251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:07.971010 16936 consensus_queue.cc:237] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:15:08.086273 11943 debug-util.cc:398] Leaking SignalData structure 0x7b08002dfc40 after lost signal to thread 16306
I20260808 20:15:08.082657 12048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 2129) took 5979 ms. Trace:
I20260808 20:15:08.084203 12241 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 655) took 1196 ms. Trace:
I20260808 20:15:08.063709 16983 raft_consensus.cc:3037] T 2cd3d6998af44e69a7182125e02f16e2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:08.065532 12249 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 657) took 1127 ms. Trace:
I20260808 20:15:08.048588 17025 leader_election.cc:288] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:08.066572 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3051f9a615344b5c80595ef0ace350c8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:08.067262 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:08.140760 17010 consensus_queue.cc:1046] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:15:07.990953 17027 consensus_queue.cc:237] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.141743 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4918de0ffe94198a66ac1975ddbc693" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:08.141866 16675 raft_consensus.cc:3032] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 8 LEADER]: Stepping down as leader of term 8
I20260808 20:15:08.142356 16675 raft_consensus.cc:729] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:15:08.031972 16853 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.005635 12036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 669) took 1113 ms. Trace:
I20260808 20:15:08.145838 12036 rpcz_store.cc:276] 0808 20:15:06.891802 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.960975 (+1069173us) service_pool.cc:227] Handling call
0808 20:15:07.961344 (+ 369us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:08.005378 (+ 44034us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:08.005576 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:08.147898 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:08.068883 16903 consensus_queue.cc:260] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.070422 16861 raft_consensus.cc:3032] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 20:15:08.149175 16861 raft_consensus.cc:729] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:15:08.150213 16861 consensus_queue.cc:260] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:08.150907 16670 raft_consensus.cc:2781] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:15:08.071513 12245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2095) took 5418 ms. Trace:
I20260808 20:15:08.073277 12039 raft_consensus.cc:1261] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260808 20:15:08.151422 16670 raft_consensus.cc:482] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:08.151727 12039 pending_rounds.cc:76] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting all ops after (but not including) 4
I20260808 20:15:08.083072 12043 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49226 (request call id 664) took 1192 ms. Trace:
I20260808 20:15:08.096084 12248 raft_consensus.cc:2454] T faebf2544dae4e21b7519aeeb8b250dc P b9d94f35ea15421aa1d721201c782d8a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 10.
I20260808 20:15:08.099642 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:08.152086 12039 pending_rounds.cc:97] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260808 20:15:08.152280 16903 raft_consensus.cc:3037] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:08.153229 12039 raft_consensus.cc:2866] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 20:15:08.154798 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:08.154939 12248 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96"
dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
mode: GRACEFUL
new_leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
from {username='slave'} at 127.0.0.1:40912
I20260808 20:15:08.099819 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:08.084033 16971 leader_election.cc:288] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
W20260808 20:15:08.100641 11943 debug-util.cc:398] Leaking SignalData structure 0x7b08002fdee0 after lost signal to thread 16501
I20260808 20:15:08.131601 12001 leader_election.cc:302] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:08.103984 12251 raft_consensus.cc:1226] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 4. Current term is 5. Ops: []
I20260808 20:15:08.104450 12101 leader_election.cc:302] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:15:08.061305 12254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53030 (request call id 653) took 1198 ms. Trace:
I20260808 20:15:08.152768 12045 raft_consensus.cc:2379] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 5: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:15:08.094483 12146 rpcz_store.cc:276] 0808 20:15:06.852925 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.926429 (+ 73504us) service_pool.cc:227] Handling call
0808 20:15:08.057817 (+1131388us) spinlock_profiling.cc:243] Waited 746 ms on lock 0x7b54003ed354. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:15:08.057968 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1714753920}
I20260808 20:15:08.152254 12043 rpcz_store.cc:276] 0808 20:15:06.890551 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:08.082069 (+1191518us) service_pool.cc:227] Handling call
0808 20:15:08.082468 (+ 399us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:08.082730 (+ 262us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:08.082756 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:08.082810 (+ 54us) raft_consensus.cc:1631] Marking committed up to 8
0808 20:15:08.082844 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:08.082897 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:08.083000 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:08.098073 12249 rpcz_store.cc:276] 0808 20:15:06.938343 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:08.064508 (+1126165us) service_pool.cc:227] Handling call
0808 20:15:08.064938 (+ 430us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:08.065225 (+ 287us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:08.065249 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:08.065288 (+ 39us) raft_consensus.cc:1631] Marking committed up to 6
0808 20:15:08.065313 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:08.065360 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:08.065463 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:08.094765 12250 rpcz_store.cc:276] 0808 20:15:04.544103 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.044722 (+2500619us) service_pool.cc:227] Handling call
0808 20:15:07.046054 (+ 1332us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:07.877672 (+831618us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:08.058570 (+180898us) spinlock_profiling.cc:243] Waited 774 ms on lock 0x7b540007e254. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:15:08.058746 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1780631424}
I20260808 20:15:08.158346 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:08.085577 16992 leader_election.cc:288] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:08.099679 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:08.158211 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:08.157272 12254 rpcz_store.cc:276] 0808 20:15:06.863328 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.889410 (+ 26082us) service_pool.cc:227] Handling call
0808 20:15:08.061100 (+1171690us) spinlock_profiling.cc:243] Waited 837 ms on lock 0x7b5400086e54. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682de2ae 00007fa96d21bc27 00007fa966903c04 00007fa966903b92 00007fa966903ac1 00007fa966903a3c 00007fa966902cf2 00007fa96670494c 00007fa966703d86 00007fa96670372f
0808 20:15:08.061238 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1925976320}
I20260808 20:15:08.160190 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:08.160429 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b217f6c1ea849f89667b8eb2d857206" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:08.160799 12152 raft_consensus.cc:3037] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:15:08.159020 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:08.161612 13014 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:15:08.087193 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:08.096205 12048 rpcz_store.cc:276] 0808 20:15:02.102828 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.628505 (+5525677us) service_pool.cc:227] Handling call
0808 20:15:07.722556 (+ 94051us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 20:15:07.833607 (+111051us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.833636 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 20:15:07.834248 (+ 612us) log.cc:842] Serialized 26 byte log entry
0808 20:15:07.909745 (+ 75497us) raft_consensus.cc:1631] Marking committed up to 4
0808 20:15:07.909781 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.909843 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:15:08.082441 (+172598us) raft_consensus.cc:1671] finished
0808 20:15:08.082461 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:08.082599 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7341,"raft.queue_time_us":10977,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"thread_start_us":75031,"threads_started":1,"wal-append.queue_time_us":156924}
I20260808 20:15:08.156834 12251 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 660) took 1172 ms. Trace:
I20260808 20:15:08.157125 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:08.162711 12251 rpcz_store.cc:276] 0808 20:15:06.984019 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:08.103467 (+1119448us) service_pool.cc:227] Handling call
0808 20:15:08.103802 (+ 335us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:08.156587 (+ 52785us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:08.156758 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:08.108562 13075 raft_consensus.cc:2454] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:15:08.152480 12245 rpcz_store.cc:276] 0808 20:15:02.652842 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:06.789870 (+4137028us) service_pool.cc:227] Handling call
0808 20:15:06.790148 (+ 278us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:08.014411 (+1224263us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:08.071214 (+ 56803us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b5400086954. stack: 00007fa965380033 00007fa96537ff30 00007fa964cde534 00007fa96e9ea8b4 00007fa96e9ea849 00007fa96825f471 00007fa9682daf51 00007fa9682d999e 00007fa96d21b2e9 00007fa966901684 00007fa966901612 00007fa966901541 00007fa9669014bc 00007fa966900772 00007fa96670494c 00007fa966703d86
0808 20:15:08.071436 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2691058048}
I20260808 20:15:08.141870 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:08.102854 12247 raft_consensus.cc:1226] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 3. Current term is 4. Ops: []
I20260808 20:15:08.161556 12250 raft_consensus.cc:3037] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 3 FOLLOWER]: Advancing to term 5
I20260808 20:15:08.164729 12245 raft_consensus.cc:1226] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6. Ops: []
I20260808 20:15:08.158274 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:08.168100 12045 raft_consensus.cc:2379] T eec183c460ad4f1b85ad1407c53f2eab P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 9: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:15:08.119432 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc15f09b4a4470ac96054b70602187" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:08.168220 16508 raft_consensus.cc:504] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.122612 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020c14c3aeb419493cd543071ed7801" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:08.182440 12150 raft_consensus.cc:2454] T e020c14c3aeb419493cd543071ed7801 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:15:08.124009 12138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:08.124483 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5943c3d789423691d3fe7099372c59" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:08.183382 12245 raft_consensus.cc:1226] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 5. Current term is 6. Ops: []
I20260808 20:15:08.125972 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:08.184242 12137 raft_consensus.cc:2379] T ac9505fd371049b7b3ca47aad176ad64 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 7: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:15:08.184819 17025 consensus_queue.cc:1046] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 20:15:08.185433 12137 raft_consensus.cc:1261] T 237b38cfed904b8395749a75666444e9 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 20:15:08.186112 16859 raft_consensus.cc:3032] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:15:08.186513 16859 raft_consensus.cc:729] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:15:08.186873 12138 raft_consensus.cc:2454] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:15:08.183351 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:08.149315 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:08.188665 12150 raft_consensus.cc:2379] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 9: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:15:08.188339 16859 consensus_queue.cc:260] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.126590 16434 raft_consensus.cc:504] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:08.127493 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e4c0cacf074dee848eed1ab4d27ebf" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:08.128769 16982 raft_consensus.cc:504] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.129160 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:15:08.169020 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 80184.5us
I20260808 20:15:08.191546 16982 leader_election.cc:288] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:08.192118 11994 leader_election.cc:302] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:15:08.131800 13072 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 9.
I20260808 20:15:08.189877 16859 raft_consensus.cc:3037] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:15:08.189271 12152 raft_consensus.cc:2454] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:15:08.133774 16706 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:08.137686 16400 raft_consensus.cc:504] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.216168 16400 leader_election.cc:288] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:08.143381 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:08.219909 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5f3a393f488ca713b7269b5dfa78" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:08.067663 16867 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.221186 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf7252c47a48cdb408ea9e49270f80" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:08.096993 12241 rpcz_store.cc:276] 0808 20:15:06.888250 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:08.031218 (+1142968us) service_pool.cc:227] Handling call
0808 20:15:08.031661 (+ 443us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:08.083929 (+ 52268us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:08.084121 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:08.151955 16670 raft_consensus.cc:3037] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:15:08.155584 12039 raft_consensus.cc:2379] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 7: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
W20260808 20:15:08.156200 11943 debug-util.cc:398] Leaking SignalData structure 0x7b080035aea0 after lost signal to thread 16542
W20260808 20:15:08.222440 11943 debug-util.cc:398] Leaking SignalData structure 0x7b08000ae620 after lost signal to thread 16574
I20260808 20:15:08.162011 16881 raft_consensus.cc:2726] T efc388dadfe84be190c9ddd0b3725244 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 20:15:08.121142 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb83ac6240244be967995e7771a8b64" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:08.158967 12043 raft_consensus.cc:3037] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:15:08.151371 16861 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Advancing to term 8
W20260808 20:15:08.164685 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 70347.5us
W20260808 20:15:08.224664 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 266457us
I20260808 20:15:08.170363 16588 raft_consensus.cc:504] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.227806 16434 leader_election.cc:288] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:08.228505 16588 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:08.164122 12251 raft_consensus.cc:1226] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 3. Current term is 4. Ops: []
I20260808 20:15:08.170487 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e931b9df2c74729a3771d5f0734f559" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:08.173477 12254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 662) took 1105 ms. Trace:
I20260808 20:15:08.155535 12248 raft_consensus.cc:595] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Received request to transfer leadership to TS 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:15:08.165120 12247 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 659) took 1188 ms. Trace:
I20260808 20:15:08.174369 16809 consensus_queue.cc:1046] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 20:15:08.168967 12147 raft_consensus.cc:2454] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 5.
I20260808 20:15:08.233472 12247 rpcz_store.cc:276] 0808 20:15:06.976728 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:08.102285 (+1125557us) service_pool.cc:227] Handling call
0808 20:15:08.102673 (+ 388us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:08.164864 (+ 62191us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:08.165068 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 20:15:08.235074 11943 debug-util.cc:398] Leaking SignalData structure 0x7b0800393aa0 after lost signal to thread 16605
W20260808 20:15:08.235378 11943 debug-util.cc:398] Leaking SignalData structure 0x7b08003e5360 after lost signal to thread 16633
I20260808 20:15:08.177093 16853 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747)
I20260808 20:15:08.105903 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73cf4b21a754e75b2af936d7402def2" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:08.177934 16901 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:08.238914 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e59eff611e4cc693407979ebe0857c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:08.241061 12101 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:15:08.244071 12101 leader_election.cc:302] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:08.245606 16881 raft_consensus.cc:2781] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:15:08.246058 16881 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:15:08.247452 16881 raft_consensus.cc:3037] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:08.247641 16809 raft_consensus.cc:3032] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:15:08.248049 16809 raft_consensus.cc:729] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:15:08.248487 17025 consensus_queue.cc:1046] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 20:15:08.248672 17042 consensus_queue.cc:1046] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260808 20:15:08.178561 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 47095us
I20260808 20:15:08.113458 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:08.169615 17033 raft_consensus.cc:2726] T eec183c460ad4f1b85ad1407c53f2eab P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 20:15:08.062610 11889 catalog_manager.cc:5495] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:15:08.181159 16864 raft_consensus.cc:504] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.161214 12249 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53030 (request call id 661) took 1152 ms. Trace:
I20260808 20:15:08.192380 12041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53028 (request call id 2126) took 6380 ms. Trace:
I20260808 20:15:08.208937 12250 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:15:08.193084 11994 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:15:08.143316 16675 consensus_queue.cc:260] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:08.222806 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d977879a1ef46db915d4b277255a08c" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:08.224188 13071 raft_consensus.cc:2379] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 8: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:15:08.225981 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:08.266860 12249 rpcz_store.cc:276] 0808 20:15:07.009103 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:08.160262 (+1151159us) service_pool.cc:227] Handling call
0808 20:15:08.160667 (+ 405us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:08.160848 (+ 181us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:08.160873 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:08.160931 (+ 58us) raft_consensus.cc:1631] Marking committed up to 8
0808 20:15:08.160998 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:08.161045 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:08.161144 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13056}
I20260808 20:15:08.233408 12248 raft_consensus.cc:603] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 20:15:08.234143 12048 raft_consensus.cc:1226] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 LEADER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 6. Current term is 7. Ops: []
I20260808 20:15:08.234305 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:08.233117 12254 rpcz_store.cc:276] 0808 20:15:07.068481 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:08.160249 (+1091768us) service_pool.cc:227] Handling call
0808 20:15:08.160612 (+ 363us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:08.160829 (+ 217us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:08.160853 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 20:15:08.160911 (+ 58us) raft_consensus.cc:1631] Marking committed up to 8
0808 20:15:08.173240 (+ 12329us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:08.173296 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:08.173397 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:08.268474 12001 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:15:08.266953 12244 raft_consensus.cc:2379] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e5bd9a4616ce4acfa20dc7a002984a20 in current term 7: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:15:08.237561 12251 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44378 (request call id 2131) took 1009 ms. Trace:
I20260808 20:15:08.269161 12251 rpcz_store.cc:276] 0808 20:15:07.222617 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:08.163558 (+940941us) service_pool.cc:227] Handling call
0808 20:15:08.163964 (+ 406us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 20:15:08.232273 (+ 68309us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:08.232439 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 20:15:08.249028 16809 consensus_queue.cc:260] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.249650 17025 raft_consensus.cc:3032] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:15:08.270452 16809 raft_consensus.cc:3037] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:15:08.270555 17025 raft_consensus.cc:729] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:15:08.271427 17025 consensus_queue.cc:260] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.267177 12041 rpcz_store.cc:276] 0808 20:15:01.812266 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 20:15:07.610138 (+5797872us) service_pool.cc:227] Handling call
0808 20:15:07.622266 (+ 12128us) raft_consensus.cc:1457] Updating replica for 3 ops
0808 20:15:07.814673 (+192407us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 20:15:07.814707 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0808 20:15:07.835979 (+ 21272us) log.cc:842] Serialized 78 byte log entry
0808 20:15:07.943752 (+107773us) raft_consensus.cc:1631] Marking committed up to 3
0808 20:15:07.999125 (+ 55373us) log.cc:842] Serialized 14 byte log entry
0808 20:15:07.999570 (+ 445us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 20:15:07.999619 (+ 49us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 20:15:08.192069 (+192450us) raft_consensus.cc:1671] finished
0808 20:15:08.192089 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 20:15:08.192229 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39080,"prepare.queue_time_us":55213,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":136950,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":5308,"thread_start_us":254775,"threads_started":2,"wal-append.queue_time_us":351525}
I20260808 20:15:08.272728 17025 raft_consensus.cc:3037] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:15:08.272173 12245 raft_consensus.cc:1261] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260808 20:15:08.274534 12245 pending_rounds.cc:76] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a: Aborting all ops after (but not including) 7
I20260808 20:15:08.274748 12245 pending_rounds.cc:97] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260808 20:15:08.274989 12245 raft_consensus.cc:2866] T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 20:15:08.250342 11943 debug-util.cc:398] Leaking SignalData structure 0x7b08001a0940 after lost signal to thread 16644
I20260808 20:15:08.209877 16982 raft_consensus.cc:2781] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 20:15:08.278036 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
W20260808 20:15:08.278944 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 85979.1us
I20260808 20:15:08.249814 17042 raft_consensus.cc:3032] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:15:08.283102 12247 raft_consensus.cc:1226] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 6. Current term is 7. Ops: []
I20260808 20:15:08.290696 12101 leader_election.cc:302] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:15:08.246768 15674 raft_consensus.cc:2726] T c0f516c855714940a41eac002203e145 P 78b9263889cd4f1582578a3aa4f644d9 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 20:15:08.293488 12001 leader_election.cc:302] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:15:08.289397 10336 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260808 20:15:08.294610 11943 debug-util.cc:398] Leaking SignalData structure 0x7b0800343e00 after lost signal to thread 16712
W20260808 20:15:08.294989 11943 debug-util.cc:398] Leaking SignalData structure 0x7b0800362b60 after lost signal to thread 16731
W20260808 20:15:08.295220 11943 debug-util.cc:398] Leaking SignalData structure 0x7b080032e260 after lost signal to thread 16734
I20260808 20:15:08.289741 16830 consensus_queue.cc:1035] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: NON_VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.002s
I20260808 20:15:08.296679 13077 raft_consensus.cc:1226] T 940aa50483d844deb4031f312b4a7355 P e5bd9a4616ce4acfa20dc7a002984a20 [term 9 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 8. Current term is 9. Ops: []
I20260808 20:15:08.275998 16477 raft_consensus.cc:686] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
W20260808 20:15:08.302188 11943 debug-util.cc:398] Leaking SignalData structure 0x7b08003f5660 after lost signal to thread 16737
W20260808 20:15:08.302485 11943 debug-util.cc:398] Leaking SignalData structure 0x7b08003f5980 after lost signal to thread 16743
W20260808 20:15:08.302655 11943 debug-util.cc:398] Leaking SignalData structure 0x7b08002197e0 after lost signal to thread 16744
I20260808 20:15:08.298874 16508 leader_election.cc:288] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:08.289764 17042 raft_consensus.cc:729] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:15:08.289145 16599 raft_consensus.cc:2781] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:15:08.309146 11994 leader_election.cc:302] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:15:08.313031 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:08.279532 16982 raft_consensus.cc:482] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:08.290282 16867 leader_election.cc:288] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:08.289264 17027 raft_consensus.cc:2781] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:15:08.305210 16864 leader_election.cc:288] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:08.305931 16901 raft_consensus.cc:2726] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 20:15:08.318439 17027 raft_consensus.cc:482] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:15:08.307036 10336 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 20:15:08.307451 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:08.308090 12142 raft_consensus.cc:2396] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260808 20:15:08.308974 11889 catalog_manager.cc:5495] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 2 to 4, leader changed from e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) to 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1). New cstate: current_term: 4 leader_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } } }
I20260808 20:15:08.309567 17042 consensus_queue.cc:260] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.305238 17031 consensus_queue.cc:1046] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
W20260808 20:15:08.303689 11943 debug-util.cc:398] Leaking SignalData structure 0x7b0800399140 after lost signal to thread 16746
I20260808 20:15:08.317795 12043 raft_consensus.cc:2454] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:15:08.318502 13074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:08.318820 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:08.321491 13077 raft_consensus.cc:2379] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 7: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:15:08.324771 13074 raft_consensus.cc:3037] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:15:08.322002 17027 raft_consensus.cc:3037] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:15:08.325634 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:08.326236 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:08.322220 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:08.327443 12038 raft_consensus.cc:1226] T 940aa50483d844deb4031f312b4a7355 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 8. Current term is 9. Ops: []
I20260808 20:15:08.328294 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ef909a266f4100bd54d95062fc7e25" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:08.329315 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:08.329862 12041 raft_consensus.cc:2379] T 7aff1371284542e48cbd467d3ef0ede9 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 7: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:15:08.324259 10336 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 20:15:08.331120 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:08.318241 16982 raft_consensus.cc:3037] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:15:08.309191 16599 raft_consensus.cc:686] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:15:08.316370 16938 consensus_queue.cc:1035] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.018s
I20260808 20:15:08.319952 16675 raft_consensus.cc:3037] T b0f8a2331e88407c9e3da24439d202b2 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:08.336781 12101 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:15:08.322474 16830 raft_consensus.cc:2781] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:15:08.336592 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:08.333267 16983 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:08.328922 16901 raft_consensus.cc:2726] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 20:15:08.346899 11943 debug-util.cc:398] Leaking SignalData structure 0x7b0800369000 after lost signal to thread 16787
I20260808 20:15:08.338944 16853 raft_consensus.cc:3032] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 LEADER]: Stepping down as leader of term 6
W20260808 20:15:08.349457 11943 debug-util.cc:398] Leaking SignalData structure 0x7b0800268940 after lost signal to thread 16798
I20260808 20:15:08.347905 12205 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:15:08.350101 11994 leader_election.cc:302] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:08.345893 16983 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.347613 17042 raft_consensus.cc:3037] T 72e59eff611e4cc693407979ebe0857c P e5bd9a4616ce4acfa20dc7a002984a20 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:15:08.352710 16983 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:08.349267 16853 raft_consensus.cc:729] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:15:08.354030 11889 catalog_manager.cc:5495] T 4e5943c3d789423691d3fe7099372c59 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 4 to 5, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 5 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:15:08.355894 11889 catalog_manager.cc:5495] T 003bc27b07fa40e48c1e57a13b578981 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 5 to 6, leader changed from e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 6 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 20:15:08.353997 16853 consensus_queue.cc:260] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.358062 16853 raft_consensus.cc:3037] T 0be6c610fe68472b8bcb6232b862dbc1 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:08.359274 17029 consensus_queue.cc:1046] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 20:15:08.360436 17031 raft_consensus.cc:3032] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 20:15:08.360800 17031 raft_consensus.cc:729] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:15:08.361795 17031 consensus_queue.cc:260] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.363210 17031 raft_consensus.cc:3037] T 102eea36004446e5a71aaccebb9ad859 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:08.343942 16599 consensus_queue.cc:237] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.349211 16830 raft_consensus.cc:686] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:15:08.368335 16830 consensus_queue.cc:237] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
W20260808 20:15:08.369616 11943 debug-util.cc:398] Leaking SignalData structure 0x7b08003efbe0 after lost signal to thread 16899
W20260808 20:15:08.370014 11943 debug-util.cc:398] Leaking SignalData structure 0x7b0800384960 after lost signal to thread 16910
W20260808 20:15:08.370299 11943 debug-util.cc:398] Leaking SignalData structure 0x7b08003da4e0 after lost signal to thread 16922
W20260808 20:15:08.370558 11943 debug-util.cc:398] Leaking SignalData structure 0x7b0800204c60 after lost signal to thread 16928
W20260808 20:15:08.370762 11943 debug-util.cc:398] Leaking SignalData structure 0x7b0800107780 after lost signal to thread 16929
W20260808 20:15:08.370942 11943 debug-util.cc:398] Leaking SignalData structure 0x7b0800385400 after lost signal to thread 16932
W20260808 20:15:08.371349 11943 debug-util.cc:398] Leaking SignalData structure 0x7b08003659a0 after lost signal to thread 16952
I20260808 20:15:08.372296 13073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:08.372939 13073 raft_consensus.cc:2454] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:15:08.373508 12205 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:08.374284 16434 raft_consensus.cc:2726] T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 20:15:08.374874 16434 consensus_queue.cc:1035] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:15:08.375780 16861 raft_consensus.cc:2781] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 20:15:08.375969 17041 consensus_queue.cc:1046] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 20:15:08.376219 16861 raft_consensus.cc:482] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:15:08.376864 16861 raft_consensus.cc:3037] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:15:08.377135 16588 raft_consensus.cc:3032] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 20:15:08.377389 16992 consensus_queue.cc:1046] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 20:15:08.377640 16588 raft_consensus.cc:729] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:15:08.376272 16434 raft_consensus.cc:2726] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:15:08.343067 16903 raft_consensus.cc:482] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:08.378566 16992 raft_consensus.cc:3032] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 8 LEADER]: Stepping down as leader of term 8
I20260808 20:15:08.378610 16588 consensus_queue.cc:260] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:08.382802 16588 raft_consensus.cc:3037] T 7aff1371284542e48cbd467d3ef0ede9 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:08.381485 16992 raft_consensus.cc:729] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:15:08.332677 16477 consensus_queue.cc:237] T 2cd3d6998af44e69a7182125e02f16e2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:08.384521 16992 consensus_queue.cc:260] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.384855 16936 consensus_queue.cc:1046] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 20:15:08.385715 16992 raft_consensus.cc:3037] T 940aa50483d844deb4031f312b4a7355 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:08.385803 16936 raft_consensus.cc:3032] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:15:08.386248 16936 raft_consensus.cc:729] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:15:08.386974 16936 consensus_queue.cc:260] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:08.388327 16936 raft_consensus.cc:3037] T 6b217f6c1ea849f89667b8eb2d857206 P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:15:08.389693 17017 consensus_queue.cc:1046] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:15:08.391723 16670 raft_consensus.cc:504] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.381472 16903 raft_consensus.cc:504] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } attrs { replace: true } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.393793 16983 raft_consensus.cc:482] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:15:08.394345 16983 raft_consensus.cc:504] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:08.395459 16903 leader_election.cc:288] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:08.397055 16983 leader_election.cc:288] T 0285ac4646ab40c3b248ff959bbcd41e P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:08.402889 16892 consensus_queue.cc:1046] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 20:15:08.404433 17041 raft_consensus.cc:3032] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 20:15:08.404767 17041 raft_consensus.cc:729] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:15:08.409400 13072 raft_consensus.cc:2454] T 2d977879a1ef46db915d4b277255a08c P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:15:08.406335 12205 leader_election.cc:302] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:08.419894 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:08.419795 12043 raft_consensus.cc:1261] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260808 20:15:08.420717 12040 raft_consensus.cc:3037] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:15:08.422770 16508 raft_consensus.cc:2726] T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 20:15:08.422942 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358aa11611bd4b16a062e6ced419bef8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:08.421211 11889 catalog_manager.cc:5495] T f73cf4b21a754e75b2af936d7402def2 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 2 to 4, leader changed from e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 4 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260808 20:15:08.424031 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 77551us
I20260808 20:15:08.417876 16867 raft_consensus.cc:482] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:08.424911 11889 catalog_manager.cc:5495] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 2 to 4, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 4 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260808 20:15:08.427624 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:08.428253 12038 raft_consensus.cc:3037] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:15:08.428938 12045 raft_consensus.cc:1226] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 6. Current term is 7. Ops: []
I20260808 20:15:08.419999 13074 raft_consensus.cc:2454] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8.
I20260808 20:15:08.424371 12041 raft_consensus.cc:1226] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 5. Current term is 6. Ops: []
I20260808 20:15:08.424995 16867 raft_consensus.cc:504] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.422608 17039 raft_consensus.cc:2726] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 20:15:08.434584 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:08.426635 11889 catalog_manager.cc:5495] T f4918de0ffe94198a66ac1975ddbc693 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 20:15:08.432400 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:08.430289 16861 raft_consensus.cc:504] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.435158 12048 raft_consensus.cc:3037] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:15:08.433437 12001 leader_election.cc:302] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:15:08.429073 11994 leader_election.cc:302] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:15:08.428408 17027 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:08.446296 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088cb69cb3c4d66afe9dc98da9179ea" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:08.443167 16864 raft_consensus.cc:3037] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:15:08.453960 12243 raft_consensus.cc:1226] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 6. Current term is 7. Ops: []
I20260808 20:15:08.446434 11889 catalog_manager.cc:5495] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 reported cstate change: term changed from 3 to 7, leader changed from b9d94f35ea15421aa1d721201c782d8a (127.10.24.3) to 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2). New cstate: current_term: 7 leader_uuid: "78b9263889cd4f1582578a3aa4f644d9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 20:15:08.440537 16508 raft_consensus.cc:2781] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 20:15:08.460860 12205 leader_election.cc:302] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:08.466822 12146 raft_consensus.cc:1226] T b0f8a2331e88407c9e3da24439d202b2 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 LEADER]: Rejecting Update request from peer b9d94f35ea15421aa1d721201c782d8a for earlier term 8. Current term is 9. Ops: []
I20260808 20:15:08.414392 16881 raft_consensus.cc:504] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:08.444033 16983 raft_consensus.cc:2726] T 72e59eff611e4cc693407979ebe0857c P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 20:15:08.437675 16938 consensus_queue.cc:1035] T 1e0d98ab3b4945d8b845711f4ddd7153 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.030s
I20260808 20:15:08.475031 12207 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
W20260808 20:15:08.475575 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 50670.8us
I20260808 20:15:08.478407 16861 leader_election.cc:288] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:08.471626 16508 raft_consensus.cc:686] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:15:08.482432 17027 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:08.440919 16765 consensus_queue.cc:1046] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260808 20:15:08.489424 12038 raft_consensus.cc:2454] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 5.
I20260808 20:15:08.491357 13015 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
W20260808 20:15:08.492936 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 85418.8us
I20260808 20:15:08.494068 16989 raft_consensus.cc:2726] T 67b75df4537844c2b76699830dedad62 P b9d94f35ea15421aa1d721201c782d8a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:15:08.497905 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
W20260808 20:15:08.498736 12000 leader_election.cc:334] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.727s (SENT)
I20260808 20:15:08.500763 12000 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:08.499090 16779 raft_consensus.cc:2781] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 20:15:08.501672 16779 raft_consensus.cc:686] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:15:08.502552 16779 consensus_queue.cc:237] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } attrs { replace: true } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.499181 12243 raft_consensus.cc:1261] T 4e393e90875847809e1ef94179c5b4ef P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 20:15:08.498999 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16032fd7a244ded8d9660c183645279" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:08.490156 16670 leader_election.cc:288] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 election: Requested vote from peers b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:08.507375 13071 raft_consensus.cc:2379] T a16032fd7a244ded8d9660c183645279 P e5bd9a4616ce4acfa20dc7a002984a20 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 5: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:15:08.508173 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cecc37ef974ec58b1d5fe18e9801a2" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:08.405598 17041 consensus_queue.cc:260] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:08.511417 13075 raft_consensus.cc:2454] T 926cdce6649c4ca59961cfac78392647 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 6.
I20260808 20:15:08.496201 16903 raft_consensus.cc:2781] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 20:15:08.515328 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:08.515928 12241 raft_consensus.cc:2454] T d092815e2e3e40ba8cbcc396cbeeb3b7 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 5.
I20260808 20:15:08.514690 12105 leader_election.cc:302] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:08.517446 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:08.517940 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:08.517472 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
W20260808 20:15:08.519666 12205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 66343.1us
I20260808 20:15:08.520239 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:08.521006 16508 consensus_queue.cc:237] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.509409 17027 raft_consensus.cc:2726] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:15:08.511494 16853 consensus_queue.cc:1035] T 4e393e90875847809e1ef94179c5b4ef P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20260808 20:15:08.525784 16765 raft_consensus.cc:3037] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:15:08.526546 16864 raft_consensus.cc:2726] T 4e5943c3d789423691d3fe7099372c59 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 20:15:08.533656 16588 raft_consensus.cc:3032] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 20:15:08.534018 16588 raft_consensus.cc:729] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
W20260808 20:15:08.535120 12000 leader_election.cc:334] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.791s (SENT)
I20260808 20:15:08.534806 16588 consensus_queue.cc:260] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:08.535559 12000 leader_election.cc:302] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:15:08.536185 16588 raft_consensus.cc:3037] T aacf7252c47a48cdb408ea9e49270f80 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:08.538813 16901 raft_consensus.cc:2726] T a9ef909a266f4100bd54d95062fc7e25 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 20:15:08.514564 16903 raft_consensus.cc:482] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 20:15:08.541280 16903 raft_consensus.cc:3037] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:08.544015 17041 raft_consensus.cc:3037] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 3 FOLLOWER]: Advancing to term 4
I20260808 20:15:08.545086 12039 raft_consensus.cc:1226] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 7. Current term is 8. Ops: []
I20260808 20:15:08.545353 12249 raft_consensus.cc:2379] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 9: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:15:08.498082 16867 leader_election.cc:288] T a16032fd7a244ded8d9660c183645279 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:08.555397 12146 raft_consensus.cc:2396] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a for term 10 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260808 20:15:08.519831 12105 leader_election.cc:302] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:08.541451 11994 leader_election.cc:302] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:15:08.547780 12205 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:08.510527 16983 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:15:08.558158 12152 raft_consensus.cc:2454] T 0285ac4646ab40c3b248ff959bbcd41e P 78b9263889cd4f1582578a3aa4f644d9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 11.
I20260808 20:15:08.556599 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:08.559947 12048 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 8.
I20260808 20:15:08.561626 13015 leader_election.cc:302] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:15:08.562271 12205 leader_election.cc:302] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:15:08.563493 12245 raft_consensus.cc:1226] T a9ef909a266f4100bd54d95062fc7e25 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 6. Current term is 7. Ops: []
I20260808 20:15:08.564630 16989 raft_consensus.cc:2726] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 20:15:08.564922 16706 raft_consensus.cc:482] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:08.565515 16706 raft_consensus.cc:504] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:08.565963 11994 leader_election.cc:302] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:15:08.566807 16867 raft_consensus.cc:2781] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 20:15:08.567298 16867 raft_consensus.cc:686] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [term 10 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:15:08.567575 16706 leader_election.cc:288] T aa97e9c654da4bcda377ef63ec1c694b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:08.568173 16881 leader_election.cc:288] T fbc85c9f50234a49833e25d612f66f96 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:08.570559 12245 raft_consensus.cc:1226] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 6. Current term is 7. Ops: []
I20260808 20:15:08.570994 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:08.571544 16971 raft_consensus.cc:2781] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:15:08.572062 16971 raft_consensus.cc:686] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:15:08.573048 16971 consensus_queue.cc:237] T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:15:08.573756 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 53997.8us
I20260808 20:15:08.574474 12105 leader_election.cc:302] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:08.575237 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aff1371284542e48cbd467d3ef0ede9" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:08.575960 12241 raft_consensus.cc:2379] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 7: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:15:08.573695 12244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
W20260808 20:15:08.577673 17052 log.cc:924] Time spent T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.064s user 0.001s sys 0.003s
I20260808 20:15:08.561718 16901 raft_consensus.cc:2781] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:15:08.579180 16901 raft_consensus.cc:482] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:08.579490 16864 raft_consensus.cc:1050] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995: attempting to promote NON_VOTER e5bd9a4616ce4acfa20dc7a002984a20 to VOTER
I20260808 20:15:08.579658 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21889249378248049ef043f0588cebc9" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:08.577710 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:08.580801 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0285ac4646ab40c3b248ff959bbcd41e" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:08.568096 16867 consensus_queue.cc:237] T faebf2544dae4e21b7519aeeb8b250dc P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:08.582973 12040 raft_consensus.cc:2454] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5bd9a4616ce4acfa20dc7a002984a20 in term 8.
I20260808 20:15:08.602576 13015 leader_election.cc:302] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
W20260808 20:15:08.601989 12102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.1:40747, user_credentials={real_user=slave}} blocked reactor thread for 251871us
I20260808 20:15:08.563253 16983 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:08.617157 12250 raft_consensus.cc:1261] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260808 20:15:08.616321 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:08.618275 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2607c6f53fc243d2abb6b2f74b6e2aa6" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:08.618276 12244 raft_consensus.cc:1226] T 85cecc37ef974ec58b1d5fe18e9801a2 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 5. Current term is 6. Ops: []
I20260808 20:15:08.610417 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a" is_pre_election: true
I20260808 20:15:08.579530 16901 raft_consensus.cc:3037] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:15:08.620822 12241 raft_consensus.cc:2379] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 5: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
W20260808 20:15:08.606729 16864 raft_consensus.cc:1054] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995: Unable to promote non-voter e5bd9a4616ce4acfa20dc7a002984a20: Illegal state: Leader has not yet committed an operation in its own term
I20260808 20:15:08.622221 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926cdce6649c4ca59961cfac78392647" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
W20260808 20:15:08.621166 13016 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 122231us
W20260808 20:15:08.628265 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 53862.7us
W20260808 20:15:08.628746 12000 leader_election.cc:334] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.819s (SENT)
I20260808 20:15:08.632432 12000 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:08.607379 17042 raft_consensus.cc:2781] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 20:15:08.634764 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b75df4537844c2b76699830dedad62" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:08.634876 12207 leader_election.cc:302] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:15:08.645264 12249 raft_consensus.cc:2379] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:15:08.670742 12243 raft_consensus.cc:2454] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 7.
I20260808 20:15:08.636157 12105 leader_election.cc:302] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:08.619848 16779 raft_consensus.cc:2781] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 20:15:08.617749 17017 raft_consensus.cc:2726] T 1d7d7a85e3964f3f9d5f2dc39e23340b P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:15:08.697346 12000 leader_election.cc:302] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
W20260808 20:15:08.698434 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 63645.5us
I20260808 20:15:08.699277 12207 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:08.606914 17029 consensus_queue.cc:1046] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.058s
I20260808 20:15:08.702558 12207 leader_election.cc:302] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:15:08.630172 16881 raft_consensus.cc:2726] T e5dc2ab0712f4054883178f4b4c50928 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 20:15:08.637791 16670 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Advancing to term 8
W20260808 20:15:08.634526 17051 log.cc:924] Time spent T 5fdbdda415914f1a9652ef68d1bba7fb P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20260808 20:15:08.715106 12245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d91cb12be034dcc9ca3038df3cc85cf" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:08.716547 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:08.726377 12241 raft_consensus.cc:3037] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:15:08.729974 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6c610fe68472b8bcb6232b862dbc1" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:08.730758 12247 raft_consensus.cc:2379] T 0be6c610fe68472b8bcb6232b862dbc1 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 7: Already voted for candidate b9d94f35ea15421aa1d721201c782d8a in this term.
I20260808 20:15:08.641067 17056 consensus_queue.cc:1046] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.066s
I20260808 20:15:08.704957 16992 raft_consensus.cc:2781] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:15:08.731978 13075 raft_consensus.cc:1261] T 237b38cfed904b8395749a75666444e9 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 20:15:08.636770 16892 consensus_queue.cc:1035] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.078s
I20260808 20:15:08.750131 12000 leader_election.cc:302] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a
W20260808 20:15:08.753294 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 117205us
I20260808 20:15:08.717217 16675 raft_consensus.cc:2781] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 20:15:08.714480 17042 raft_consensus.cc:686] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:15:08.706326 16971 raft_consensus.cc:2726] T 21889249378248049ef043f0588cebc9 P b9d94f35ea15421aa1d721201c782d8a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 20:15:08.747611 12249 raft_consensus.cc:1203] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[3.3-7.5] Dedup: 3.3->[5.4-7.5]
I20260808 20:15:08.726135 16864 raft_consensus.cc:2726] T 5e931b9df2c74729a3771d5f0734f559 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 20:15:08.826661 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:08.725360 17017 raft_consensus.cc:2726] T c4dc15f09b4a4470ac96054b70602187 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 20:15:08.779170 12101 leader_election.cc:302] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:15:08.797504 12243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:08.830154 12249 raft_consensus.cc:1226] T 4e5943c3d789423691d3fe7099372c59 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 7. Current term is 8. Ops: [3.3-7.5]
I20260808 20:15:08.758302 11994 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:08.803429 12247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:08.836347 12247 raft_consensus.cc:2379] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 5: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:15:08.838752 17010 consensus_queue.cc:1035] T 237b38cfed904b8395749a75666444e9 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260808 20:15:08.841022 12206 leader_election.cc:334] T 6088cb69cb3c4d66afe9dc98da9179ea P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.503s (SENT)
W20260808 20:15:08.841770 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 56559us
I20260808 20:15:08.732635 16779 raft_consensus.cc:686] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 LEADER]: Becoming Leader. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:15:08.845713 12105 leader_election.cc:302] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:08.780191 16881 raft_consensus.cc:3032] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 20:15:08.749382 15674 raft_consensus.cc:3032] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:15:08.845515 16992 raft_consensus.cc:686] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
W20260808 20:15:08.849737 12101 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 70657.9us
I20260808 20:15:08.855538 16864 raft_consensus.cc:2726] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:15:08.780295 16830 raft_consensus.cc:2781] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 20:15:08.858438 16830 raft_consensus.cc:482] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting leader election (detected failure of leader b9d94f35ea15421aa1d721201c782d8a)
I20260808 20:15:08.860044 16830 raft_consensus.cc:3037] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:08.869439 16983 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139)
I20260808 20:15:08.858884 12040 raft_consensus.cc:1226] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Rejecting Update request from peer e5bd9a4616ce4acfa20dc7a002984a20 for earlier term 4. Current term is 5. Ops: []
I20260808 20:15:08.871678 16859 consensus_queue.cc:1046] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 20:15:08.871886 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:08.872725 16859 raft_consensus.cc:3032] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 20:15:08.873762 16859 raft_consensus.cc:729] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:15:08.859131 16765 raft_consensus.cc:2726] T 4e393e90875847809e1ef94179c5b4ef P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:15:08.814248 16903 raft_consensus.cc:504] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.876407 12248 raft_consensus.cc:2354] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 6. Current term is 7.
I20260808 20:15:08.877358 16903 leader_election.cc:288] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:08.875435 16859 consensus_queue.cc:260] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.881556 16992 consensus_queue.cc:237] T 2d977879a1ef46db915d4b277255a08c P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.900831 13076 raft_consensus.cc:1261] T 4e4bf0005d9e43c1a4adce7588e7653f P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260808 20:15:08.903347 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 56407.8us
I20260808 20:15:08.832392 16675 raft_consensus.cc:686] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [term 6 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:15:08.914017 12101 leader_election.cc:302] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:15:08.898862 15674 raft_consensus.cc:729] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
W20260808 20:15:08.915632 12206 leader_election.cc:334] T 7aff1371284542e48cbd467d3ef0ede9 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 1.619s (SENT)
I20260808 20:15:08.912518 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:08.920701 13078 raft_consensus.cc:3037] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:08.779443 16599 consensus_queue.cc:1035] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.128s
I20260808 20:15:08.901851 16477 raft_consensus.cc:2726] T 102eea36004446e5a71aaccebb9ad859 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 20:15:08.852087 16881 raft_consensus.cc:729] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:15:08.890713 17017 raft_consensus.cc:2781] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260808 20:15:08.909391 17061 log.cc:924] Time spent T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20260808 20:15:08.919348 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d871fe214ff9bbeacaa32fd5136b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:15:08.926945 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 168742us
I20260808 20:15:08.884675 16830 raft_consensus.cc:504] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:15:08.940465 12105 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 111522us
I20260808 20:15:08.907464 16936 consensus_queue.cc:1046] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260808 20:15:08.941342 12105 leader_election.cc:302] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:15:08.941531 17066 log.cc:924] Time spent T 3051f9a615344b5c80595ef0ace350c8 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.115s user 0.000s sys 0.000s
I20260808 20:15:08.941859 16936 raft_consensus.cc:3032] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 20:15:08.942279 16936 raft_consensus.cc:729] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:15:08.943658 16861 raft_consensus.cc:2726] T e256c29dec964f49856cf204f950793b P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 20:15:08.917842 16809 raft_consensus.cc:3032] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 20:15:08.945022 16809 raft_consensus.cc:729] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e5bd9a4616ce4acfa20dc7a002984a20, State: Running, Role: LEADER
I20260808 20:15:08.945835 16809 consensus_queue.cc:260] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.943261 12105 leader_election.cc:302] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:08.947383 16588 raft_consensus.cc:2726] T 2607c6f53fc243d2abb6b2f74b6e2aa6 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:15:08.943087 16936 consensus_queue.cc:260] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:08.948325 12101 leader_election.cc:302] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:15:08.930429 17017 raft_consensus.cc:482] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:15:08.947003 16809 raft_consensus.cc:3037] T 0d91cb12be034dcc9ca3038df3cc85cf P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:15:08.949030 17017 raft_consensus.cc:3037] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:15:08.948678 16936 raft_consensus.cc:3037] T a9ef909a266f4100bd54d95062fc7e25 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:08.955149 12105 leader_election.cc:302] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20; no voters:
I20260808 20:15:08.940544 16830 leader_election.cc:288] T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 9 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:08.937270 11994 leader_election.cc:302] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:15:08.956064 16861 raft_consensus.cc:2781] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 20:15:08.956535 16830 raft_consensus.cc:2726] T aa97e9c654da4bcda377ef63ec1c694b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 20:15:08.956576 16861 raft_consensus.cc:686] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 LEADER]: Becoming Leader. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:15:08.958053 16861 consensus_queue.cc:237] T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.958920 16588 raft_consensus.cc:2781] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 20:15:08.959291 16588 raft_consensus.cc:482] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4cd46f13bceb4c3c9ef71e755e17b995)
I20260808 20:15:08.959626 16588 raft_consensus.cc:3037] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Advancing to term 8
I20260808 20:15:08.961917 11994 leader_election.cc:302] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
W20260808 20:15:08.965734 17068 log.cc:924] Time spent T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20: Append to log took a long time: real 0.131s user 0.000s sys 0.000s
I20260808 20:15:08.932600 17042 consensus_queue.cc:237] T 7cb737cc041b47a88cab93a3e6ed8158 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:08.926304 16599 consensus_queue.cc:1225] T 21889249378248049ef043f0588cebc9 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer b9d94f35ea15421aa1d721201c782d8a log is divergent from this leader: its last log entry 5.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260808 20:15:08.987358 12000 leader_election.cc:302] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: 78b9263889cd4f1582578a3aa4f644d9, b9d94f35ea15421aa1d721201c782d8a
I20260808 20:15:08.988690 16901 raft_consensus.cc:504] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:08.969400 11994 leader_election.cc:302] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:08.938122 17039 raft_consensus.cc:2726] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 20:15:08.992264 12241 raft_consensus.cc:2454] T fbc85c9f50234a49833e25d612f66f96 P b9d94f35ea15421aa1d721201c782d8a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 8.
W20260808 20:15:08.995371 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 165799us
I20260808 20:15:09.003782 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97e9c654da4bcda377ef63ec1c694b" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:08.997859 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:09.015350 12251 raft_consensus.cc:1261] T cb8b79bb1c59483e93af813653a73a31 P b9d94f35ea15421aa1d721201c782d8a [term 5 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260808 20:15:09.020457 13078 raft_consensus.cc:2454] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 7.
I20260808 20:15:08.994377 16830 raft_consensus.cc:2726] T 926cdce6649c4ca59961cfac78392647 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 20:15:09.021867 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "b9d94f35ea15421aa1d721201c782d8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:09.023811 12207 leader_election.cc:302] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20; no voters: 4cd46f13bceb4c3c9ef71e755e17b995
I20260808 20:15:09.026279 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:09.026558 13072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20"
I20260808 20:15:09.027745 13078 raft_consensus.cc:2379] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 8: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:15:09.028713 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:09.029508 12039 raft_consensus.cc:2379] T ac9505fd371049b7b3ca47aad176ad64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 7: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:15:09.007741 16670 raft_consensus.cc:2726] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 20:15:09.031665 12249 raft_consensus.cc:1226] T 358aa11611bd4b16a062e6ced419bef8 P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 6. Current term is 7. Ops: []
I20260808 20:15:09.033581 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:09.034322 13071 raft_consensus.cc:2379] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 8: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
I20260808 20:15:09.036276 13076 raft_consensus.cc:1226] T 358aa11611bd4b16a062e6ced419bef8 P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 6. Current term is 7. Ops: []
I20260808 20:15:09.036816 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:08.989619 16477 consensus_queue.cc:1046] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20260808 20:15:09.043061 16477 raft_consensus.cc:3032] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 20:15:09.043381 16477 raft_consensus.cc:729] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:15:09.044128 16477 consensus_queue.cc:260] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
W20260808 20:15:09.027498 17072 log.cc:924] Time spent T 1e0d98ab3b4945d8b845711f4ddd7153 P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.109s user 0.001s sys 0.000s
I20260808 20:15:09.045472 16477 raft_consensus.cc:3037] T 85cecc37ef974ec58b1d5fe18e9801a2 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
W20260808 20:15:09.045783 17067 log.cc:924] Time spent T 4b5f5f3a393f488ca713b7269b5dfa78 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.230s user 0.001s sys 0.003s
W20260808 20:15:09.039429 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 70103.1us
I20260808 20:15:09.041285 16670 raft_consensus.cc:2726] T 6b217f6c1ea849f89667b8eb2d857206 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 20:15:09.036511 16830 raft_consensus.cc:2726] T 67b75df4537844c2b76699830dedad62 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:15:09.048367 11994 leader_election.cc:302] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:09.047129 16867 consensus_queue.cc:1035] T 4e4bf0005d9e43c1a4adce7588e7653f P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
W20260808 20:15:09.050998 17041 consensus_meta.cc:287] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9: Time spent flushing consensus metadata: real 0.507s user 0.005s sys 0.003s
I20260808 20:15:09.053253 11994 leader_election.cc:302] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:09.057279 12105 leader_election.cc:302] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 78b9263889cd4f1582578a3aa4f644d9; no voters: b9d94f35ea15421aa1d721201c782d8a, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:09.057893 12045 raft_consensus.cc:2379] T 78ef4041ff114e34a9dff167914c95ef P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 9: Already voted for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in this term.
W20260808 20:15:09.068604 12207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 44879us
I20260808 20:15:09.068636 16903 raft_consensus.cc:2781] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election won for term 7
I20260808 20:15:09.070040 16903 raft_consensus.cc:686] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 7 LEADER]: Becoming Leader. State: Replica: b9d94f35ea15421aa1d721201c782d8a, State: Running, Role: LEADER
I20260808 20:15:09.070946 16903 consensus_queue.cc:237] T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:09.071769 16588 raft_consensus.cc:504] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:09.073040 12146 raft_consensus.cc:2379] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d94f35ea15421aa1d721201c782d8a in current term 8: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:15:09.073896 16588 leader_election.cc:288] T a83f6daa15a54982865040c0110bdda8 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 8 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:15:09.074604 12001 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.4:36901, user_credentials={real_user=slave}} blocked reactor thread for 72771us
W20260808 20:15:09.075037 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 85941.5us
I20260808 20:15:09.024667 16881 consensus_queue.cc:260] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:09.028483 15674 consensus_queue.cc:260] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:09.088184 15674 raft_consensus.cc:3037] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Advancing to term 6
I20260808 20:15:09.089257 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6f3bbe5e92475487be203c7c89b73d" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" is_pre_election: true
I20260808 20:15:09.075681 16830 raft_consensus.cc:2726] T 0d91cb12be034dcc9ca3038df3cc85cf P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 20:15:09.080582 16670 raft_consensus.cc:2726] T 0be6c610fe68472b8bcb6232b862dbc1 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260808 20:15:09.079900 17070 log.cc:924] Time spent T 9d6f3bbe5e92475487be203c7c89b73d P 4cd46f13bceb4c3c9ef71e755e17b995: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
I20260808 20:15:09.084574 16881 raft_consensus.cc:3037] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
W20260808 20:15:09.084982 13014 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 45777.1us
I20260808 20:15:09.085472 16853 consensus_queue.cc:1035] T cb8b79bb1c59483e93af813653a73a31 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 20:15:09.089200 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83f6daa15a54982865040c0110bdda8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:09.022572 16675 consensus_queue.cc:237] T 926cdce6649c4ca59961cfac78392647 P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:09.082526 16936 raft_consensus.cc:2726] T cb8b79bb1c59483e93af813653a73a31 P 78b9263889cd4f1582578a3aa4f644d9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 20:15:09.111891 12142 raft_consensus.cc:2454] T 6088cb69cb3c4d66afe9dc98da9179ea P 78b9263889cd4f1582578a3aa4f644d9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d94f35ea15421aa1d721201c782d8a in term 4.
I20260808 20:15:09.116050 13072 raft_consensus.cc:1226] T e912d871fe214ff9bbeacaa32fd5136b P e5bd9a4616ce4acfa20dc7a002984a20 [term 7 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 6. Current term is 7. Ops: []
I20260808 20:15:09.082536 16765 consensus_queue.cc:1046] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.017s
W20260808 20:15:09.089267 16892 consensus_queue.cc:1159] T 5fdbdda415914f1a9652ef68d1bba7fb P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" responder_term: 6 status { last_received { term: 6 index: 7 } last_committed_idx: 7 last_received_current_leader { term: 6 index: 7 } }
W20260808 20:15:09.105216 16982 consensus_meta.cc:287] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995: Time spent flushing consensus metadata: real 0.762s user 0.005s sys 0.005s
I20260808 20:15:09.106277 16901 leader_election.cc:288] T 3051f9a615344b5c80595ef0ace350c8 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
W20260808 20:15:09.106640 17059 log.cc:924] Time spent T 1e0d98ab3b4945d8b845711f4ddd7153 P b9d94f35ea15421aa1d721201c782d8a: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260808 20:15:09.117350 11994 leader_election.cc:398] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 election: Vote denied by peer b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391) with higher term. Message: Invalid argument: T e912d871fe214ff9bbeacaa32fd5136b P b9d94f35ea15421aa1d721201c782d8a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 6. Current term is 7.
W20260808 20:15:09.137477 11994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.3:38391, user_credentials={real_user=slave}} blocked reactor thread for 60492.1us
I20260808 20:15:09.012693 16779 consensus_queue.cc:237] T fd0cef71281c44f6b3fd2afe876735f8 P e5bd9a4616ce4acfa20dc7a002984a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:09.146446 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41bae2134ac343ed9e9f9c1b86703d97" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:09.147449 12038 raft_consensus.cc:3037] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:09.148010 11994 leader_election.cc:302] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd46f13bceb4c3c9ef71e755e17b995, b9d94f35ea15421aa1d721201c782d8a; no voters:
I20260808 20:15:09.151276 12039 raft_consensus.cc:1226] T f4918de0ffe94198a66ac1975ddbc693 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Rejecting Update request from peer 78b9263889cd4f1582578a3aa4f644d9 for earlier term 6. Current term is 7. Ops: []
I20260808 20:15:09.163352 12038 raft_consensus.cc:2454] T 41bae2134ac343ed9e9f9c1b86703d97 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b9263889cd4f1582578a3aa4f644d9 in term 9.
I20260808 20:15:09.164492 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
W20260808 20:15:09.169651 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 182393us
I20260808 20:15:09.130069 16982 raft_consensus.cc:504] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:09.179344 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0cef71281c44f6b3fd2afe876735f8" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:09.179749 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003bc27b07fa40e48c1e57a13b578981" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:09.182911 12045 raft_consensus.cc:2379] T fd0cef71281c44f6b3fd2afe876735f8 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 8: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:15:09.187570 17093 raft_consensus.cc:482] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 20:15:09.188165 17093 raft_consensus.cc:504] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:09.189042 16982 leader_election.cc:288] T 6088cb69cb3c4d66afe9dc98da9179ea P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 4 election: Requested vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:09.189399 13071 raft_consensus.cc:1261] T 003bc27b07fa40e48c1e57a13b578981 P e5bd9a4616ce4acfa20dc7a002984a20 [term 8 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260808 20:15:09.182340 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8b79bb1c59483e93af813653a73a31" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:09.190243 17093 leader_election.cc:288] T 5fdbdda415914f1a9652ef68d1bba7fb P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), e5bd9a4616ce4acfa20dc7a002984a20 (127.10.24.4:36901)
I20260808 20:15:09.190550 12039 raft_consensus.cc:2379] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b9263889cd4f1582578a3aa4f644d9 in current term 5: Already voted for candidate e5bd9a4616ce4acfa20dc7a002984a20 in this term.
I20260808 20:15:09.191813 12048 raft_consensus.cc:1261] T 4b5f5f3a393f488ca713b7269b5dfa78 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260808 20:15:09.169919 16901 raft_consensus.cc:2781] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 FOLLOWER]: Leader election won for term 8
W20260808 20:15:09.197134 12000 leader_election.cc:334] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139): Timed out: RequestConsensusVote RPC to 127.10.24.2:35139 timed out after 2.146s (SENT)
I20260808 20:15:09.170363 17062 consensus_queue.cc:1046] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260808 20:15:09.166374 16864 consensus_queue.cc:1046] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260808 20:15:09.209695 12048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc85c9f50234a49833e25d612f66f96" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cd46f13bceb4c3c9ef71e755e17b995"
I20260808 20:15:09.197701 16989 consensus_queue.cc:1035] T 9d6f3bbe5e92475487be203c7c89b73d P b9d94f35ea15421aa1d721201c782d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20260808 20:15:09.216638 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ef4041ff114e34a9dff167914c95ef" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:09.217306 12142 raft_consensus.cc:3037] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 8 FOLLOWER]: Advancing to term 9
I20260808 20:15:09.218261 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb737cc041b47a88cab93a3e6ed8158" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:09.218889 12147 raft_consensus.cc:2454] T 7cb737cc041b47a88cab93a3e6ed8158 P 78b9263889cd4f1582578a3aa4f644d9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 7.
I20260808 20:15:09.231204 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc388dadfe84be190c9ddd0b3725244" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:09.204680 17017 raft_consensus.cc:504] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:09.233044 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce978b5a06de44eabba785178c7a0100" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:09.233822 12146 raft_consensus.cc:2379] T ce978b5a06de44eabba785178c7a0100 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 in current term 6: Already voted for candidate 78b9263889cd4f1582578a3aa4f644d9 in this term.
I20260808 20:15:09.235028 12150 raft_consensus.cc:1261] T 21889249378248049ef043f0588cebc9 P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260808 20:15:09.235423 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d092815e2e3e40ba8cbcc396cbeeb3b7" candidate_uuid: "78b9263889cd4f1582578a3aa4f644d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b9d94f35ea15421aa1d721201c782d8a"
I20260808 20:15:09.233062 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd3d6998af44e69a7182125e02f16e2" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:09.233242 12141 raft_consensus.cc:1226] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Rejecting Update request from peer 4cd46f13bceb4c3c9ef71e755e17b995 for earlier term 5. Current term is 6. Ops: []
I20260808 20:15:09.237012 12206 leader_election.cc:302] T f73cf4b21a754e75b2af936d7402def2 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 78b9263889cd4f1582578a3aa4f644d9, e5bd9a4616ce4acfa20dc7a002984a20
I20260808 20:15:09.249661 13075 raft_consensus.cc:1261] T ce978b5a06de44eabba785178c7a0100 P e5bd9a4616ce4acfa20dc7a002984a20 [term 6 FOLLOWER]: Refusing update from remote peer 4cd46f13bceb4c3c9ef71e755e17b995: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 20:15:09.247237 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdbdda415914f1a9652ef68d1bba7fb" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" is_pre_election: true
I20260808 20:15:09.248183 12142 raft_consensus.cc:2396] T 78ef4041ff114e34a9dff167914c95ef P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cd46f13bceb4c3c9ef71e755e17b995 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260808 20:15:09.254665 12152 raft_consensus.cc:1261] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Refusing update from remote peer b9d94f35ea15421aa1d721201c782d8a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260808 20:15:09.255134 12152 pending_rounds.cc:76] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9: Aborting all ops after (but not including) 2
I20260808 20:15:09.255357 12152 pending_rounds.cc:97] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260808 20:15:09.255601 12152 raft_consensus.cc:2866] T 4b5f5f3a393f488ca713b7269b5dfa78 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 20:15:09.261597 17092 log.cc:924] Time spent T 41bae2134ac343ed9e9f9c1b86703d97 P 78b9263889cd4f1582578a3aa4f644d9: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20260808 20:15:09.262831 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a383bcab83014009b5698c63cdac298b" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9" is_pre_election: true
I20260808 20:15:09.263422 12150 raft_consensus.cc:2454] T a383bcab83014009b5698c63cdac298b P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 6.
I20260808 20:15:09.264083 17081 raft_consensus.cc:482] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 78b9263889cd4f1582578a3aa4f644d9)
I20260808 20:15:09.264598 17081 raft_consensus.cc:504] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } }
I20260808 20:15:09.266634 17081 leader_election.cc:288] T ddb83ac6240244be967995e7771a8b64 P 4cd46f13bceb4c3c9ef71e755e17b995 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78b9263889cd4f1582578a3aa4f644d9 (127.10.24.2:35139), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:09.267397 16830 consensus_queue.cc:1035] T 003bc27b07fa40e48c1e57a13b578981 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 20:15:09.271111 16859 raft_consensus.cc:3037] T 67b75df4537844c2b76699830dedad62 P e5bd9a4616ce4acfa20dc7a002984a20 [term 4 FOLLOWER]: Advancing to term 5
I20260808 20:15:09.278724 16670 raft_consensus.cc:3032] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 20:15:09.279099 16670 raft_consensus.cc:729] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:15:09.279847 16670 consensus_queue.cc:260] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:09.281915 16588 raft_consensus.cc:3032] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 20:15:09.282294 16588 raft_consensus.cc:729] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 78b9263889cd4f1582578a3aa4f644d9, State: Running, Role: LEADER
I20260808 20:15:09.283061 16588 consensus_queue.cc:260] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78b9263889cd4f1582578a3aa4f644d9" member_type: VOTER last_known_addr { host: "127.10.24.2" port: 35139 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:09.284129 16588 raft_consensus.cc:3037] T 358aa11611bd4b16a062e6ced419bef8 P 78b9263889cd4f1582578a3aa4f644d9 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:09.284233 12039 raft_consensus.cc:1261] T cb8b79bb1c59483e93af813653a73a31 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Refusing update from remote peer e5bd9a4616ce4acfa20dc7a002984a20: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260808 20:15:09.286306 17106 consensus_queue.cc:1035] T ce978b5a06de44eabba785178c7a0100 P 4cd46f13bceb4c3c9ef71e755e17b995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 20:15:09.295562 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec183c460ad4f1b85ad1407c53f2eab" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
I20260808 20:15:09.296128 12152 raft_consensus.cc:2362] T eec183c460ad4f1b85ad1407c53f2eab P 78b9263889cd4f1582578a3aa4f644d9 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4cd46f13bceb4c3c9ef71e755e17b995 in term 9. Re-sending same reply.
I20260808 20:15:09.246124 17017 leader_election.cc:288] T d092815e2e3e40ba8cbcc396cbeeb3b7 P 78b9263889cd4f1582578a3aa4f644d9 [CANDIDATE]: Term 6 election: Requested vote from peers 4cd46f13bceb4c3c9ef71e755e17b995 (127.10.24.1:40747), b9d94f35ea15421aa1d721201c782d8a (127.10.24.3:38391)
I20260808 20:15:09.248981 16901 raft_consensus.cc:686] T fbc85c9f50234a49833e25d612f66f96 P 4cd46f13bceb4c3c9ef71e755e17b995 [term 8 LEADER]: Becoming Leader. State: Replica: 4cd46f13bceb4c3c9ef71e755e17b995, State: Running, Role: LEADER
I20260808 20:15:09.322611 17081 raft_consensus.cc:482] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e5bd9a4616ce4acfa20dc7a002984a20)
I20260808 20:15:09.323141 17081 raft_consensus.cc:504] T 2d977879a1ef46db915d4b277255a08c P 4cd46f13bceb4c3c9ef71e755e17b995 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" member_type: VOTER last_known_addr { host: "127.10.24.1" port: 40747 } } peers { permanent_uuid: "b9d94f35ea15421aa1d721201c782d8a" member_type: VOTER last_known_addr { host: "127.10.24.3" port: 38391 } } peers { permanent_uuid: "e5bd9a4616ce4acfa20dc7a002984a20" member_type: VOTER last_known_addr { host: "127.10.24.4" port: 36901 } attrs { promote: false } }
I20260808 20:15:09.307132 16670 raft_consensus.cc:3037] T e912d871fe214ff9bbeacaa32fd5136b P 4cd46f13bceb4c3c9ef71e755e17b995 [term 6 FOLLOWER]: Advancing to term 7
I20260808 20:15:09.329137 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faebf2544dae4e21b7519aeeb8b250dc" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"
W20260808 20:15:09.334347 12206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.24.2:35139, user_credentials={real_user=slave}} blocked reactor thread for 138810us
I20260808 20:15:09.335201 12206 leader_election.cc:302] T 6b217f6c1ea849f89667b8eb2d857206 P b9d94f35ea15421aa1d721201c782d8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d94f35ea15421aa1d721201c782d8a; no voters: 4cd46f13bceb4c3c9ef71e755e17b995, 78b9263889cd4f1582578a3aa4f644d9
I20260808 20:15:09.337455 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9505fd371049b7b3ca47aad176ad64" candidate_uuid: "4cd46f13bceb4c3c9ef71e755e17b995" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "78b9263889cd4f1582578a3aa4f644d9"